Video encoder, video decoder, methods for encoding and decoding and video data stream for realizing advanced video coding concepts

ABSTRACT

An apparatus (200) for receiving an input video data stream according to an embodiment is provided. The input video data stream has a video encoded thereinto. The apparatus (200) is configured to generate an output video data stream from the input video data stream.

The present invention relates to video encoding and video decoding and,in particular, to a video encoder, to a video decoder, to methods forencoding and decoding and to a video data stream for realizing advancedvideo coding concepts.

H.265/HEVC (HEVC=High Efficiency Video Coding) is a video codec whichalready provides tools for elevating or even enabling parallelprocessing at an encoder and/or at a decoder. For example, HEVC supportsa sub-division of pictures into an array of tiles which are encodedindependently from each other. Another concept supported by HEVCpertains to WPP, according to which CTU-rows or CTU-lines of thepictures may be processed in parallel from left to right, e.g. instripes, provided that some minimum CTU offset is obeyed in theprocessing of consecutive CTU lines (CTU=coding tree unit). It would befavorable, however, to have a video codec at hand which supportsparallel processing capabilities of video encoders and/or video decoderseven more efficiently.

In the following, an introduction to VCL partitioning according to thestate-of-the-art is described (VCL=video coding layer).

Typically, in video coding, a coding process of picture samples requiressmaller partitions, where samples are divided into some rectangularareas for joint processing such as prediction or transform coding.Therefore, a picture is partitioned into blocks of a particular sizethat is constant during encoding of the video sequence. In H.264/AVCstandard fixed-size blocks of 16×16 samples, so called macroblocks, areused (AVC=Advanced Video Coding).

In the state-of-the-art HEVC standard (see [1]), there are Coded TreeBlocks (CTB) or Coding Tree Units (CTU) of a maximum size of 64×64samples. In the further description of HEVC, for such a kind of blocks,the more common term CTU is used.

CTUs are processed in raster scan order, starting with the top-left CTU,processing CTUs in the picture line-wise, down to the bottom-right CTU.

The coded CTU data is organized into a kind of container called slice.Originally, in former video coding standards, slice means a segmentcomprising one or more consecutive CTUs of a picture. Slices areemployed for a segmentation of coded data. From another point of view,the complete picture can also be defined as one big segment and hence,historically, the term slice is still applied. Besides the coded picturesamples, slices also comprise additional information related to thecoding process of the slice itself which is placed into a so-calledslice header.

According to the state-of-the-art, a VCL (video coding layer) alsocomprises techniques for fragmentation and spatial partitioning. Suchpartitioning may, e.g., be applied in video coding for various reasons,among which are processing load-balancing in parallelization, CTU sizematching in network transmission, error-mitigation etc.

Other examples relate to RoI (RoI=Region of Interest) encodings, wherethere is for example a region in the middle of the picture that viewerscan select e.g. with a zoom in operation (decoding only the RoI), orgradual decoder refresh (GDR) in which intra data (that is typically putinto one frame of a video sequence) is temporally distributed overseveral successive frames, e.g. as a column of intra blocks that swipesover the picture plane and resets the temporal prediction chain locallyin the same fashion as an intra picture does it for the whole pictureplane. For the latter, two regions exist in each picture, one that isrecently reset and one that is potentially affected by errors and errorpropagation.

Reference Picture Resampling (RPR) is a technique used in video codingto adapt the quality/rate of the video not only by using a coarserquantization parameter but by adapting the resolution of potentiallyeach transmitted picture. Thus, references used for inter predictionmight have a different size that the picture that is currently beingpredicted for encoding. Basically, RPR requires a resampling process inthe prediction loop, e.g., upsampling and downsampling filters to bedefined.

Depending on flavor, RPR can result in a change of coded picture size atany picture, or be limited to happen at only some particular picture,e.g. only at particular positions bounded for instance to segmentboundaries adaptive HTTP streaming.

The object of the present invention is to provide improved concepts forvideo encoding and video decoding.

The object of the present invention is solved by the subject-matter ofthe independent claims.

In accordance with a first aspect of the invention, an apparatus forreceiving an input video data stream is provided. The input video datastream has a video encoded thereinto. The apparatus is configured togenerate an output video data stream from the input video data stream.Moreover, the apparatus is to determine whether a picture of the videopreceding a dependent random access picture shall be output or not.

Moreover, a video data stream is provided. The video data stream has avideo encoded thereinto. The video data stream comprises an indicationthat indicates whether a picture of the video preceding a dependentrandom access picture shall be output or not.

Furthermore, a video encoder is provided. The video encoder isconfigured to encode a video into a video data stream. Moreover, thevideo encoder is configured to generate the video data stream such thatthe video data stream comprises an indication that indicates whether apicture of the video preceding a dependent random access picture shallbe output or not.

Moreover, a video decoder for receiving a video data stream having avideo stored therein is provided. The video decoder is configured todecode the video from the video data stream. The video decoder isconfigured to decode the video depending on an indication indicatingwhether a picture of the video preceding a dependent random accesspicture shall be output or not.

Furthermore, a method for receiving an input video data stream isprovided. The input video data stream has a video encoded thereinto. Themethod comprises generating an output video data stream from the inputvideo data stream. Moreover, the method comprises determining whether apicture of the video preceding a dependent random access picture shallbe output or not.

Moreover, a method for encoding a video into a video data stream isprovided. The method comprises generating the video data stream suchthat the video data stream comprises an indication that indicateswhether a picture of the video preceding a dependent random accesspicture shall be output or not.

Furthermore, a method for receiving a video data stream having a videostored therein is provided. The method comprises decoding the video fromthe video data stream. Decoding the video is conducted depending on anindication indicating whether a picture of the video preceding adependent random access picture shall be output or not.

Moreover, computer programs for implementing one of the above-describedmethods when being executed on a computer or signal processor areprovided.

In accordance with a second aspect of the invention, an apparatus forreceiving one or more input video data streams is provided. Each of theone or more input video data streams has an input video encodedthereinto. The apparatus is configured to generate an output video datastream from the one or more input video data streams, the output videodata stream encoding an output video, wherein the apparatus isconfigured to generate the output video data stream such that the outputvideo is the input video being encoded within one of the one or moreinput video data streams, or such that the output video depends on theinput video of at least one of the one or more input video data streams.Moreover, the apparatus is configured to determine an access unitremoval time of a current picture of a plurality of pictures of theoutput video from a coded picture buffer. The apparatus is configured todetermine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture from the coded picture buffer.

Furthermore, a video data stream is provided. The video data stream hasa video encoded thereinto. The video data stream comprises coded picturebuffer delay offset information.

Furthermore, a video decoder for receiving a video data stream a videostored therein is provided. The video decoder is configured to decodethe video from the video data stream. Moreover, the video decoder isconfigured to decode the video depending on an access unit removal timeof a current picture of a plurality of pictures of the video from acoded picture buffer. The video decoder is configured to decode thevideo depending on an indication indicating whether or not to use codedpicture buffer delay offset information for determining the access unitremoval time of the current picture from the coded picture buffer.

Moreover, a method for receiving one or more input video data streams isprovided. Each of the one or more input video data streams has an inputvideo encoded thereinto. The method comprises generating an output videodata stream from the one or more input video data streams, the outputvideo data stream encoding an output video, wherein generating theoutput video data stream is conducted such that the output video is theinput video being encoded within one of the one or more input video datastreams, or such that the output video depends on the input video of atleast one of the one or more input video data streams. Moreover, themethod comprises determining an access unit removal time of a currentpicture of a plurality of pictures of the output video from a codedpicture buffer. Furthermore, the method comprises determining whether ornot to use coded picture buffer delay offset information for determiningthe access unit removal time of the current picture from the codedpicture buffer.

Furthermore, a method for encoding a video into a video data streamaccording to an embodiment is provided. The method comprises generatingthe video data stream such that the video data stream comprises codedpicture buffer delay offset information.

Moreover, a method for receiving a video data stream a video storedtherein is provided. The method comprises decoding the video from thevideo data stream. Decoding the video is conducted depending on anaccess unit removal time of a current picture of a plurality of picturesof the video from a coded picture buffer. Moreover, decoding the videois conducted depending on an indication indicating whether or not to usecoded picture buffer delay offset information for determining the accessunit removal time of the current picture from the coded picture buffer.

Furthermore, computer programs for implementing one of theabove-described methods when being executed on a computer or signalprocessor are provided.

In accordance with a third aspect of the invention, a video data streamis provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises an initial coded picturebuffer removal delay. Furthermore, the video data stream comprises aninitial coded picture buffer removal offset. Moreover, the video datastream comprises information that indicates whether or not a sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset is defined to be constant across two or morebuffering periods.

Furthermore, a video encoder is provided. The video encoder isconfigured to encode a video into a video data stream. Moreover, thevideo encoder is configured to generate the video data stream such thatthe video data stream comprises an initial coded picture buffer removaldelay. Furthermore, the video encoder is configured to generate thevideo data stream such that the video data stream comprises an initialcoded picture buffer removal offset. Moreover, the video encoder isconfigured to generate the video data stream such that the video datastream comprises information that indicates whether or not a sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset is defined to be constant across two or morebuffering periods.

Moreover, an apparatus for receiving two input video data streams, beinga first input video data stream and a second input video data stream, isprovided. Each of the two input video data streams has an input videoencoded thereinto. The apparatus is configured to generate an outputvideo data stream from the two input video data streams, the outputvideo data stream encoding an output video, wherein the apparatus isconfigured to generate an output video data stream by concatenating thefirst input video data stream and the second input video data stream.Moreover, the apparatus is configured to generate the output video datastream such that the output video data stream comprises an initial codedpicture buffer removal delay. Furthermore, the apparatus is configuredto generate the output video data stream such that the output video datastream comprises an initial coded picture buffer removal offset.Moreover, the apparatus is configured to generate the output video datastream such that the output video data stream comprises information thatindicates whether or not a sum of the initial coded picture bufferremoval delay and the initial coded picture buffer removal offset isdefined to be constant across two or more buffering periods.

Furthermore, a video decoder for receiving a video data stream a videostored therein is provided. The video decoder is configured to decodethe video from the video data stream. Moreover, the video data streamcomprises an initial coded picture buffer removal delay. Furthermore,the video data stream comprises an initial coded picture buffer removaloffset. Moreover, the video data stream comprises information thatindicates whether or not a sum of the initial coded picture bufferremoval delay and the initial coded picture buffer removal offset isdefined to be constant across two or more buffering periods.Furthermore, the video decoder is configured to decode the videodepending on the information that indicates whether or not the sum ofthe initial coded picture buffer removal delay and the initial codedpicture buffer removal offset is defined to be constant across the twoor more buffering periods.

Moreover, a method for encoding a video into a video data stream isprovided. The method comprises generating the video data stream suchthat the video data stream comprises an initial coded picture bufferremoval delay. Furthermore, the method comprises generating the videodata stream such that the video data stream comprises an initial codedpicture buffer removal offset. Moreover, the method comprises generatingthe video data stream such that the video data stream comprisesinformation that indicates whether or not a sum of the initial codedpicture buffer removal delay and the initial coded picture bufferremoval offset is defined to be constant across two or more bufferingperiods.

Furthermore, a method for receiving two input video data streams, beinga first input video data stream and a second input video data stream, isprovided. Each of the two input video data streams has an input videoencoded thereinto. The method comprises generating an output video datastream from the two input video data streams, the output video datastream encoding an output video, wherein the apparatus is configured togenerate an output video data stream by concatenating the first inputvideo data stream and the second input video data stream. Moreover, themethod comprises generating the output video data stream such that theoutput video data stream comprises an initial coded picture bufferremoval delay. Furthermore, the method comprises generating the outputvideo data stream such that the output video data stream comprises aninitial coded picture buffer removal offset. Moreover, the methodcomprises generating the output video data stream such that the outputvideo data stream comprises information that indicates whether or not asum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant across twoor more buffering periods.

Moreover, a method for receiving a video data stream a video storedtherein is provided. The method comprises decoding the video from thevideo data stream. The video data stream comprises an initial codedpicture buffer removal delay. Moreover, the video data stream comprisesan initial coded picture buffer removal offset. Furthermore, the videodata stream comprises information that indicates whether or not a sum ofthe initial coded picture buffer removal delay and the initial codedpicture buffer removal offset is defined to be constant across two ormore buffering periods. The method comprises decoding the videodepending on the information that indicates whether or not the sum ofthe initial coded picture buffer removal delay and the initial codedpicture buffer removal offset is defined to be constant across the twoor more buffering periods.

Furthermore, computer programs for implementing one of theabove-described methods when being executed on a computer or signalprocessor are provided.

In accordance with a fourth aspect of the invention, a video data streamis provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises an indication indicatingwhether or not a non-scalable nested picture timing supplementalenhancement information message of a network abstraction layer unit ofan access unit of the plurality of access units of a coded videosequence of a one or more coded video sequences of the video data streamis defined to apply to all output layer sets of a plurality of outputlayer sets of said access unit. If the indication has a first value,then the non-scalable nested picture timing supplemental enhancementinformation message of said network abstraction layer unit of saidaccess unit is defined to apply to all output layer sets of theplurality of output layer sets of said access unit. If the indicationhas a value being different from the first value, then the indicationdoes not define whether or not the non-scalable nested picture timingsupplemental enhancement information message of said network abstractionlayer unit of said access unit applies to all output layer sets of theplurality of output layer sets of said access unit.

Moreover, a video encoder is provided. The video encoder is configuredto encode a video into a video data stream. Moreover, the video encoderis configured to generate the video data stream such that the video datastream comprises an indication indicating whether or not a non-scalablenested picture timing supplemental enhancement information message of anetwork abstraction layer unit of an access unit of the plurality ofaccess units of a coded video sequence of a one or more coded videosequences of the video data stream is defined to apply to all outputlayer sets of a plurality of output layer sets of said access unit. Ifthe indication has a first value, then the non-scalable nested picturetiming supplemental enhancement information message of said networkabstraction layer unit of said access unit is defined to apply to alloutput layer sets of the plurality of output layer sets of said accessunit. If the indication has a value being different from the firstvalue, then the indication does not define whether or not thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unitapplies to all output layer sets of the plurality of output layer setsof said access unit.

Furthermore, an apparatus for receiving an input video data stream isprovided. The input video data stream has a video encoded thereinto. Theapparatus is configured to generate an processed video data stream fromthe input video data stream. Moreover, the apparatus is configured togenerate the processed video data stream such that the processed videodata stream comprises an indication indicating whether or not anon-scalable nested picture timing supplemental enhancement informationmessage of a network abstraction layer unit of an access unit of theplurality of access units of a coded video sequence of a one or morecoded video sequences of the processed video data stream is defined toapply to all output layer sets of a plurality of output layer sets ofsaid access unit. If the indication has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication has a value beingdifferent from the first value, then the indication does not definewhether or not the non-scalable nested picture timing supplementalenhancement information message of said network abstraction layer unitof said access unit applies to all output layer sets of the plurality ofoutput layer sets of said access unit.

Moreover, a video decoder for receiving a video data stream having avideo stored therein is provided. The video decoder is configured todecode the video from the video data stream. The video data streamcomprises an indication indicating whether or not a non-scalable nestedpicture timing supplemental enhancement information message of a networkabstraction layer unit of an access unit of the plurality of accessunits of a coded video sequence of a one or more coded video sequencesof the video data stream is defined to apply to all output layer sets ofa plurality of output layer sets of said access unit. If the indicationhas a first value, then the non-scalable nested picture timingsupplemental enhancement information message of said network abstractionlayer unit of said access unit is defined to apply to all output layersets of the plurality of output layer sets of said access unit. If theindication has a value being different from the first value, then theindication does not define whether or not the non-scalable nestedpicture timing supplemental enhancement information message of saidnetwork abstraction layer unit of said access unit applies to all outputlayer sets of the plurality of output layer sets of said access unit.

Furthermore, a method for encoding a video into a video data stream isprovided. The method comprises generating the video data stream suchthat the video data stream comprises an indication indicating whether ornot a non-scalable nested picture timing supplemental enhancementinformation message of a network abstraction layer unit of an accessunit of the plurality of access units of a coded video sequence of a oneor more coded video sequences of the video data stream is defined toapply to all output layer sets of a plurality of output layer sets ofsaid access unit. If the indication has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication has a value beingdifferent from the first value, then the indication does not definewhether or not the non-scalable nested picture timing supplementalenhancement information message of said network abstraction layer unitof said access unit applies to all output layer sets of the plurality ofoutput layer sets of said access unit.

Moreover, a method for receiving an input video data stream is provided.The input video data stream has a video encoded thereinto. The methodcomprises generating an processed video data stream from the input videodata stream. Moreover, the method comprises generating the processedvideo data stream such that the processed video data stream comprises anindication indicating whether or not a non-scalable nested picturetiming supplemental enhancement information message of a networkabstraction layer unit of an access unit of the plurality of accessunits of a coded video sequence of a one or more coded video sequencesof the processed video data stream is defined to apply to all outputlayer sets of a plurality of output layer sets of said access unit. Ifthe indication has a first value, then the non-scalable nested picturetiming supplemental enhancement information message of said networkabstraction layer unit of said access unit is defined to apply to alloutput layer sets of the plurality of output layer sets of said accessunit. If the indication has a value being different from the firstvalue, then the indication does not define whether or not thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unitapplies to all output layer sets of the plurality of output layer setsof said access unit.

Furthermore, a method for receiving a video data stream having a videostored therein is provided. The method comprises decoding the video fromthe video data stream. The video data stream comprises an indicationindicating whether or not a non-scalable nested picture timingsupplemental enhancement information message of a network abstractionlayer unit of an access unit of the plurality of access units of a codedvideo sequence of a one or more coded video sequences of the video datastream is defined to apply to all output layer sets of a plurality ofoutput layer sets of said access unit. If the indication has a firstvalue, then the non-scalable nested picture timing supplementalenhancement information message of said network abstraction layer unitof said access unit is defined to apply to all output layer sets of theplurality of output layer sets of said access unit. If the indicationhas a value being different from the first value, then the indicationdoes not define whether or not the non-scalable nested picture timingsupplemental enhancement information message of said network abstractionlayer unit of said access unit applies to all output layer sets of theplurality of output layer sets of said access unit.

Moreover, computer programs for implementing one of the above-describedmethods when being executed on a computer or signal processor areprovided.

In accordance with a fifth aspect of the invention, a video data streamis provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises one or more scalable nestedsupplemental enhancement information messages. The one or more scalablenested supplemental enhancement information messages comprise aplurality of syntax elements. Each syntax element of one or more syntaxelements of the plurality of syntax elements is defined to have a samesize in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of a portion of thevideo data stream.

Moreover, a video encoder is provided. The video encoder is configuredto encode a video into a video data stream. Moreover, the video encoderis configured to generate the video data stream such that the video datastream comprises one or more scalable nested supplemental enhancementinformation messages. Furthermore, the video encoder is configured togenerate the video data stream such that the one or more scalable nestedsupplemental enhancement information messages comprise a plurality ofsyntax elements. Moreover, the video encoder is configured to generatethe video data stream such that each syntax element of one or moresyntax elements of the plurality of syntax elements is defined to have asame size in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of a portion of thevideo data stream.

Furthermore, an apparatus for receiving an input video data stream isprovided. The input video data stream has a video encoded thereinto. Theapparatus is configured to generate an output video data stream from theinput video data stream. The video data stream comprises one or morescalable nested supplemental enhancement information messages. The oneor more scalable nested supplemental enhancement information messagescomprise a plurality of syntax elements. Each syntax element of one ormore syntax elements of the plurality of syntax elements is defined tohave a same size in every one of the scalable nested supplementalenhancement information messages of the video data stream or of aportion of the video data stream. The apparatus is configured to processthe one or more scalable nested supplemental enhancement informationmessages.

Moreover, a video decoder for receiving a video data stream having avideo stored therein is provided. The video decoder is configured todecode the video from the video data stream. The video data streamcomprises one or more scalable nested supplemental enhancementinformation messages. The one or more scalable nested supplementalenhancement information messages comprise a plurality of syntaxelements. Each syntax element of one or more syntax elements of theplurality of syntax elements is defined to have a same size in every oneof the scalable nested supplemental enhancement information messages ofthe video data stream or of a portion of the video data stream. Thevideo decoder is configured to decode the video depending on the one ormore syntax elements of the plurality of syntax elements.

Furthermore, a method for encoding a video into a video data stream isprovided. The method comprises generating the video data stream suchthat the video data stream comprises one or more scalable nestedsupplemental enhancement information messages. Moreover, the methodcomprises generating the video data stream such that the one or morescalable nested supplemental enhancement information messages comprise aplurality of syntax elements. Furthermore, the method comprisesgenerating the video data stream such that each syntax element of one ormore syntax elements of the plurality of syntax elements is defined tohave a same size in every one of the scalable nested supplementalenhancement information messages of the video data stream or of aportion of the video data stream.

Moreover, a method for receiving an input video data stream is provided.The input video data stream has a video encoded thereinto. The methodcomprises generating an output video data stream from the input videodata stream. The video data stream comprises one or more scalable nestedsupplemental enhancement information messages. The one or more scalablenested supplemental enhancement information messages comprise aplurality of syntax elements. Each syntax element of one or more syntaxelements of the plurality of syntax elements is defined to have a samesize in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of a portion of thevideo data stream. The method comprises processing the one or morescalable nested supplemental enhancement information messages.

Furthermore, a method for receiving a video data stream having a videostored therein is provided. The method comprises decoding the video fromthe video data stream. The video data stream comprises one or morescalable nested supplemental enhancement information messages. The oneor more scalable nested supplemental enhancement information messagescomprise a plurality of syntax elements. Each syntax element of one ormore syntax elements of the plurality of syntax elements is defined tohave a same size in every one of the scalable nested supplementalenhancement information messages of the video data stream or of aportion of the video data stream. Decoding the video is conducteddepending on the one or more syntax elements of the plurality of syntaxelements.

Moreover, computer programs for implementing one of the above-describedmethods when being executed on a computer or signal processor areprovided.

Preferred embodiments are provided in the dependent claims.

In the following, embodiments of the present invention are described indetail with reference to the figures, in which:

FIG. 1 illustrates a video encoder for encoding a video into a videodata stream according to an embodiment.

FIG. 2 illustrates an apparatus for receiving an input video data streamaccording to an embodiment.

FIG. 3 illustrates a video decoder for receiving a video data streamhaving a video stored therein according to an embodiment.

FIG. 4 illustrates an original bitstream (depicted at the top of FIG. 4), and a bitstream after dropping pictures (depicted at the bottom ofFIG. 4 ) according to an embodiment.

FIG. 5 illustrates a splicing of two bitstreams after pictures have beendropped from the one of the two bitstreams according to an embodiment.

FIG. 6 illustrates a splicing of two bitstreams according to anotherembodiment.

FIG. 7 illustrates two sets of HRD SEIs, scalable nested SEIs andnon-scalable nested SEIs, in a two-layer bitstream according to anembodiment.

FIG. 8 illustrates a video encoder.

FIG. 9 illustrates a video decoder.

FIG. 10 illustrates the relationship between the reconstructed signal,e.g., the reconstructed picture, on the one hand, and the combination ofthe prediction residual signal as signaled in the data stream, and theprediction signal, on the other hand.

The following description of the figures starts with a presentation of adescription of an encoder and a decoder of a block-based predictivecodec for coding pictures of a video in order to form an example for acoding framework into which embodiments of the present invention may bebuilt in. The respective encoder and decoder are described with respectto FIG. 8 to FIG. 10 . Thereinafter the description of embodiments ofthe concept of the present invention is presented along with adescription as to how such concepts could be built into the encoder anddecoder of FIG. 8 and FIG. 9 , respectively, although the embodimentsdescribed with FIG. 1 to FIG. 3 and following, may also be used to formencoders and decoders not operating according to the coding frameworkunderlying the encoder and decoder of FIG. 8 and FIG. 9 .

FIG. 8 shows a video encoder, an apparatus for predictively coding apicture 12 into a data stream 14 exemplarily using transform-basedresidual coding. The apparatus, or encoder, is indicated using referencesign 10. FIG. 9 shows a corresponding video decoder 20, e.g., anapparatus 20 configured to predictively decode the picture 12′ from thedata stream 14 also using transform-based residual decoding, wherein theapostrophe has been used to indicate that the picture 12′ asreconstructed by the decoder 20 deviates from picture 12 originallyencoded by apparatus 10 in terms of coding loss introduced by aquantization of the prediction residual signal. FIG. 8 and FIG. 9exemplarily use transform based prediction residual coding, althoughembodiments of the present application are not restricted to this kindof prediction residual coding. This is true for other details describedwith respect to FIG. 8 and FIG. 9 , too, as will be outlinedhereinafter.

The encoder 10 is configured to subject the prediction residual signalto spatial-to-spectral transformation and to encode the predictionresidual signal, thus obtained, into the data stream 14. Likewise, thedecoder 20 is configured to decode the prediction residual signal fromthe data stream 14 and subject the prediction residual signal thusobtained to spectral-to-spatial transformation.

Internally, the encoder 10 may comprise a prediction residual signalformer 22 which generates a prediction residual 24 so as to measure adeviation of a prediction signal 26 from the original signal, e.g., fromthe picture 12. The prediction residual signal former 22 may, forinstance, be a subtractor which subtracts the prediction signal from theoriginal signal, e.g., from the picture 12. The encoder 10 then furthercomprises a transformer 28 which subjects the prediction residual signal24 to a spatial-to-spectral transformation to obtain a spectral-domainprediction residual signal 24′ which is then subject to quantization bya quantizer 32, also comprised by the encoder 10. The thus quantizedprediction residual signal 24″ is coded into bitstream 14. To this end,encoder 10 may optionally comprise an entropy coder 34 which entropycodes the prediction residual signal as transformed and quantized intodata stream 14. The prediction signal 26 is generated by a predictionstage 36 of encoder 10 on the basis of the prediction residual signal24″ encoded into, and decodable from, data stream 14. To this end, theprediction stage 36 may internally, as is shown in FIG. 8 , comprise adequantizer 38 which dequantizes prediction residual signal 24″ so as togain spectral-domain prediction residual signal 24′″, which correspondsto signal 24′ except for quantization loss, followed by an inversetransformer 40 which subjects the latter prediction residual signal 24′″to an inverse transformation, e.g., a spectral-to-spatialtransformation, to obtain prediction residual signal 24′″, whichcorresponds to the original prediction residual signal 24 except forquantization loss. A combiner 42 of the prediction stage 36 thenrecombines, such as by addition, the prediction signal 26 and theprediction residual signal 24″″ so as to obtain a reconstructed signal46, e.g., a reconstruction of the original signal 12. Reconstructedsignal 46 may correspond to signal 12′. A prediction module 44 ofprediction stage 36 then generates the prediction signal 26 on the basisof signal 46 by using, for instance, spatial prediction, e.g.,intra-picture prediction, and/or temporal prediction, e.g.,inter-picture prediction.

Likewise, decoder 20, as shown in FIG. 9 , may be internally composed ofcomponents corresponding to, and interconnected in a mannercorresponding to, prediction stage 36. In particular, entropy decoder 50of decoder 20 may entropy decode the quantized spectral-domainprediction residual signal 24″ from the data stream, whereupondequantizer 52, inverse transformer 54, combiner 56 and predictionmodule 58, interconnected and cooperating in the manner described abovewith respect to the modules of prediction stage 36, recover thereconstructed signal on the basis of prediction residual signal 24″ sothat, as shown in FIG. 9 , the output of combiner 56 results in thereconstructed signal, namely picture 12′.

Although not specifically described above, it is readily clear that theencoder 10 may set some coding parameters including, for instance,prediction modes, motion parameters and the like, according to someoptimization scheme such as, for instance, in a manner optimizing somerate and distortion related criterion, e.g., coding cost. For example,encoder 10 and decoder 20 and the corresponding modules 44, 58,respectively, may support different prediction modes such asintra-coding modes and inter-coding modes. The granularity at whichencoder and decoder switch between these prediction mode types maycorrespond to a subdivision of picture 12 and 12′, respectively, intocoding segments or coding blocks. In units of these coding segments, forinstance, the picture may be subdivided into blocks being intra-codedand blocks being inter-coded. Intra-coded blocks are predicted on thebasis of a spatial, already coded/decoded neighborhood of the respectiveblock as is outlined in more detail below. Several intra-coding modesmay exist and be selected for a respective intra-coded segment includingdirectional or angular intra-coding modes according to which therespective segment is filled by extrapolating the sample values of theneighborhood along a certain direction which is specific for therespective directional intra-coding mode, into the respectiveintra-coded segment. The intra-coding modes may, for instance, alsocomprise one or more further modes such as a DC coding mode, accordingto which the prediction for the respective intra-coded block assigns aDC value to all samples within the respective intra-coded segment,and/or a planar intra-coding mode according to which the prediction ofthe respective block is approximated or determined to be a spatialdistribution of sample values described by a two-dimensional linearfunction over the sample positions of the respective intra-coded blockwith driving tilt and offset of the plane defined by the two-dimensionallinear function on the basis of the neighboring samples. Comparedthereto, inter-coded blocks may be predicted, for instance, temporally.For inter-coded blocks, motion vectors may be signaled within the datastream, the motion vectors indicating the spatial displacement of theportion of a previously coded picture of the video to which picture 12belongs, at which the previously coded/decoded picture is sampled inorder to obtain the prediction signal for the respective inter-codedblock. This means, in addition to the residual signal coding comprisedby data stream 14, such as the entropy-coded transform coefficientlevels representing the quantized spectral-domain prediction residualsignal 24″, data stream 14 may have encoded thereinto coding modeparameters for assigning the coding modes to the various blocks,prediction parameters for some of the blocks, such as motion parametersfor inter-coded segments, and optional further parameters such asparameters for controlling and signaling the subdivision of picture 12and 12′, respectively, into the segments. The decoder 20 uses theseparameters to subdivide the picture in the same manner as the encoderdid, to assign the same prediction modes to the segments, and to performthe same prediction to result in the same prediction signal.

FIG. 10 illustrates the relationship between the reconstructed signal,e.g., the reconstructed picture 12′, on the one hand, and thecombination of the prediction residual signal 24″″ as signaled in thedata stream 14, and the prediction signal 26, on the other hand. Asalready denoted above, the combination may be an addition. Theprediction signal 26 is illustrated in FIG. 10 as a subdivision of thepicture area into intra-coded blocks which are illustratively indicatedusing hatching, and inter-coded blocks which are illustrativelyindicated not-hatched. The subdivision may be any subdivision, such as aregular subdivision of the picture area into rows and columns of squareblocks or non-square blocks, or a multi-tree subdivision of picture 12from a tree root block into a plurality of leaf blocks of varying size,such as a quadtree subdivision or the like, wherein a mixture thereof isillustrated in FIG. 10 in which the picture area is first subdividedinto rows and columns of tree root blocks which are then furthersubdivided in accordance with a recursive multi-tree subdivisioning intoone or more leaf blocks.

Again, data stream 14 may have an intra-coding mode coded thereinto forintra-coded blocks 80, which assigns one of several supportedintra-coding modes to the respective intra-coded block 80. Forinter-coded blocks 82, the data stream 14 may have one or more motionparameters coded thereinto. Generally speaking, inter-coded blocks 82are not restricted to being temporally coded. Alternatively, inter-codedblocks 82 may be any block predicted from previously coded portionsbeyond the current picture 12 itself, such as previously coded picturesof a video to which picture 12 belongs, or picture of another view or anhierarchically lower layer in the case of encoder and decoder beingscalable encoders and decoders, respectively.

The prediction residual signal 24″″ in FIG. 10 is also illustrated as asubdivision of the picture area into blocks 84. These blocks might becalled transform blocks in order to distinguish same from the codingblocks 80 and 82. In effect, FIG. 10 illustrates that encoder 10 anddecoder 20 may use two different subdivisions of picture 12 and picture12′, respectively, into blocks, namely one subdivisioning into codingblocks 80 and 82, respectively, and another subdivision into transformblocks 84. Both subdivisions might be the same, e.g., each coding block80 and 82, may concurrently form a transform block 84, but FIG. 10illustrates the case where, for instance, a subdivision into transformblocks 84 forms an extension of the subdivision into coding blocks 80,82 so that any border between two blocks of blocks 80 and 82 overlays aborder between two blocks 84, or alternatively speaking each block 80,82 either coincides with one of the transform blocks 84 or coincideswith a cluster of transform blocks 84. However, the subdivisions mayalso be determined or selected independent from each other so thattransform blocks 84 could alternatively cross block borders betweenblocks 80, 82. As far as the subdivision into transform blocks 84 isconcerned, similar statements are thus true as those brought forwardwith respect to the subdivision into blocks 80, 82, e.g., the blocks 84may be the result of a regular subdivision of picture area into blocks(with or without arrangement into rows and columns), the result of arecursive multi-tree subdivisioning of the picture area, or acombination thereof or any other sort of blockation. Just as an aside,it is noted that blocks 80, 82 and 84 are not restricted to being ofquadratic, rectangular or any other shape.

FIG. 10 further illustrates that the combination of the predictionsignal 26 and the prediction residual signal 24″″ directly results inthe reconstructed signal 12′. However, it should be noted that more thanone prediction signal 26 may be combined with the prediction residualsignal 24″″ to result into picture 12′ in accordance with alternativeembodiments.

In FIG. 10 , the transform blocks 84 shall have the followingsignificance. Transformer 28 and inverse transformer 54 perform theirtransformations in units of these transform blocks 84. For instance,many codecs use some sort of DST or DCT for all transform blocks 84.Some codecs allow for skipping the transformation so that, for some ofthe transform blocks 84, the prediction residual signal is coded in thespatial domain directly. However, in accordance with embodimentsdescribed below, encoder 10 and decoder 20 are configured in such amanner that they support several transforms. For example, the transformssupported by encoder 10 and decoder 20 could comprise:

-   -   DCT-II (or DCT-III), where DCT stands for Discrete Cosine        Transform    -   DST-IV, where DST stands for Discrete Sine Transform    -   DCT-IV    -   DST-VII    -   Identity Transformation (IT)

Naturally, while transformer 28 would support all of the forwardtransform versions of these transforms, the decoder 20 or inversetransformer 54 would support the corresponding backward or inverseversions thereof:

-   -   Inverse DCT-II (or inverse DCT-III)    -   Inverse DST-IV    -   Inverse DCT-IV    -   Inverse DST-VII    -   Identity Transformation (IT)

The subsequent description provides more details on which transformscould be supported by encoder 10 and decoder 20. In any case, it shouldbe noted that the set of supported transforms may comprise merely onetransform such as one spectral-to-spatial or spatial-to-spectraltransform.

As already outlined above, FIG. 8 to FIG. 10 have been presented as anexample where the inventive concept described further below may beimplemented in order to form specific examples for encoders and decodersaccording to the present application. Insofar, the encoder and decoderof FIG. 8 and FIG. 9 , respectively, may represent possibleimplementations of the encoders and decoders described herein below.FIG. 8 and FIG. 9 are, however, only examples. An encoder according toembodiments of the present application may, however, perform block-basedencoding of a picture 12 using the concept outlined in more detail belowand being different from the encoder of FIG. 8 such as, for instance, inthat same is no video encoder, but a still picture encoder, in that samedoes not support inter-prediction, or in that the sub-division intoblocks 80 is performed in a manner different than exemplified in FIG. 10. Likewise, decoders according to embodiments of the present applicationmay perform block-based decoding of picture 12′ from data stream 14using the coding concept further outlined below, but may differ, forinstance, from the decoder 20 of FIG. 9 in that same is no videodecoder, but a still picture decoder, in that same does not supportintra-prediction, or in that same sub-divides picture 12′ into blocks ina manner different than described with respect to FIG. 10 and/or in thatsame does not derive the prediction residual from the data stream 14 intransform domain, but in spatial domain, for instance.

FIG. 1 illustrates a video encoder 100 for encoding a video into a videodata stream according to an embodiment. The video encoder 100 isconfigured to generate the video data stream such that the video datastream comprises an indication that indicates whether a picture of thevideo preceding a dependent random access picture shall be output ornot.

FIG. 2 illustrates an apparatus 200 for receiving an input video datastream according to an embodiment. The input video data stream has avideo encoded thereinto. The apparatus 200 is configured to generate anoutput video data stream from the input video data stream.

FIG. 3 illustrates a video decoder 300 for receiving a video data streamhaving a video stored therein according to an embodiment. The videodecoder 300 is configured to decode the video from the video datastream. The video decoder 300 is configured to decode the videodepending on an indication indicating whether a picture of the videopreceding a dependent random access picture shall be output or not.

Moreover, a system according to an embodiment is provided. The systemcomprises the apparatus of FIG. 2 and the video decoder of FIG. 3 . Thevideo decoder 300 of FIG. 3 is configured to receive the output videodata stream of the apparatus of FIG. 2 . The video decoder 300 of FIG. 3is configured to decode the video from the output video data stream ofthe apparatus 200 of FIG. 2 .

In an embodiment, the system may, e.g., further comprise a video encoder100 of FIG. 1 . The apparatus 200 of FIG. 2 may, e.g., be configured toreceive the video data stream from the video encoder 100 of FIG. 1 asthe input video data stream.

The (optional) intermediate device 210 of the apparatus 200 may, e.g.,be configured to receive the video data stream from the video encoder100 as an input video data stream and to generate an output video datastream from the input video data stream. For example, the intermediatedevice may, e.g., be configured to modify (header/meta data) informationof the input video data stream and/or may, e.g., be configured to deletepictures from the input video data stream and/or may configured tomix/splice the input video data stream with an additional secondbitstream having a second video encoded thereinto.

The (optional) video decoder 221 may, e.g., be configured to decode thevideo from the output video data stream.

The (optional) Hypothetical Reference Decoder 222 may, e.g., beconfigured to determine timing information for the video depending onthe output video data stream, or may, e.g., be configured to determinebuffer information for a buffer into which the video or a portion of thevideo is to be stored.

The system comprises the video encoder 101 of FIG. 1 and the videodecoder 151 of FIG. 2 .

The video encoder 101 is configured to generate the encoded videosignal. The video decoder 151 is configured to decode the encoded videosignal to reconstruct the picture of the video.

A first aspect of the invention is claimed in claims 1 to 38.

A second aspect of the invention is claimed in claims 39 to 78.

A third aspect of the invention is claimed in claims 79 to 108.

A fourth aspect of the invention is claimed in claims 109 to 134.

A fifth aspect of the invention is claimed in claims 135 to 188.

In the following, the first aspect of the invention is now described indetail.

In accordance with the first aspect of the invention, an apparatus 200for receiving an input video data stream is provided. The input videodata stream has a video encoded thereinto. The apparatus 200 isconfigured to generate an output video data stream from the input videodata stream. Moreover, the apparatus 200 is to determine whether apicture of the video preceding a dependent random access picture shallbe output or not.

According to an embodiment, the apparatus 200 may, e.g., be configuredto determine a first variable (e.g., a NoOutputBeforeDrapFlag)indicating whether the picture of the video that precedes the dependentrandom access picture shall be output or not.

In an embodiment, the apparatus 200 may, e.g., be configured to generatethe output video data stream such that the output video data stream may,e.g., comprise an indication that may, e.g., indicate whether thepicture of the video preceding the dependent random access picture shallbe output or not.

According to an embodiment, the apparatus 200 may, e.g., be configuredto generate the output video data stream such that the output video datastream may, e.g., comprise supplemental enhancement informationcomprising the indication that may, e.g., indicate whether the pictureof the video preceding the dependent random access picture shall beoutput or not.

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The apparatus 200 may, e.g., be configured to generate theoutput video data stream such that the output video data stream may,e.g., comprise a flag (e.g., a ph_pic_output_flag) having a predefinedvalue (e.g., 0) in a picture header of the independent random accesspicture, such that the predefined value (e.g., 0) of the flag (e.g., aph_pic_output_flag) may, e.g., indicate for the independent randomaccess picture directly precedes said dependent random access picturewithin the video data stream, and that said independent random accesspicture shall not be output.

According to an embodiment, the flag may, e.g., be a first flag, whereinthe apparatus 200 may, e.g., be configured to generate the output videodata stream such that the output video data stream may, e.g., comprise afurther flag in a picture parameter set of the video data stream,wherein the further flag may, e.g., indicate whether or not the firstflag (e.g., a ph_pic_output_flag) exists in the picture header of theindependent random access picture.

In an embodiment, the apparatus 200 may, e.g., be configured to generatethe output video data stream such that the output video data stream may,e.g., comprise as the indication that may, e.g., indicate whether thepicture of the video preceding the dependent random access picture shallbe output or not a supplemental enhancement information flag within asupplemental enhancement information of the output video data stream, ora picture parameter set flag within a picture parameter set of theoutput video data stream, or a sequence parameter set flag within asequence parameter set of the output video data stream, or an externalmeans flag, wherein a value of the external means flag may, e.g., be setby an external unit being external to the apparatus 200.

According to an embodiment, the apparatus 200 may, e.g., be configuredto determine a value of a second variable (e.g., a PictureOutputFlag)for the picture of the video that precedes the dependent random accesspicture depending on the first variable (e.g., aNoOutputBeforeDrapFlag), wherein the second variable (e.g., aPictureOutputFlag) may, e.g., indicate for said picture whether saidpicture shall be output or not, and wherein the apparatus 200 may, e.g.,be configured to output or to not output said picture depending on thesecond variable (e.g., a PictureOutputFlag).

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The first variable (e.g., a NoOutputBeforeDrapFlag) may, e.g.,indicate that the independent random access picture shall not be output.

According to an embodiment, the picture of the video that precedes thedependent random access picture may, e.g., be an independent randomaccess picture. The apparatus 200 may, e.g., be configured to set thefirst variable (e.g., a NoOutputBeforeDrapFlag) such that the firstvariable (e.g., a NoOutputBeforeDrapFlag) may, e.g., indicate that theindependent random access picture shall be output.

In an embodiment, the apparatus 200 may, e.g., be configured to signalto a video decoder 300, whether a picture of the video preceding adependent random access picture shall be output or not.

Moreover, a video data stream is provided. The video data stream has avideo encoded thereinto. The video data stream comprises an indicationthat indicates whether a picture of the video preceding a dependentrandom access picture shall be output or not.

According to an embodiment, the video data stream may, e.g., comprisesupplemental enhancement information comprising the indication that may,e.g., indicate whether the picture of the video preceding the dependentrandom access picture shall be output or not.

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The video data stream may, e.g., comprise a flag (e.g., aph_pic_output_flag) having a predefined value (e.g., 0) in a pictureheader of the independent random access picture, such that thepredefined value (e.g., 0) of the flag (e.g., a ph_pic_output_flag) may,e.g., indicate for the independent random access picture directlyprecedes said dependent random access picture within the video datastream, and that said independent random access picture shall not beoutput.

According to an embodiment, the flag may, e.g., be a first flag, whereinthe video data stream may, e.g., comprise a further flag in a pictureparameter set of the video data stream, wherein the further flag may,e.g., indicate whether or not the first flag (e.g., aph_pic_output_flag) exists in the picture header of the independentrandom access picture.

In an embodiment, the video data stream may, e.g., comprise as theindication that may, e.g., indicate whether the picture of the videopreceding the dependent random access picture shall be output or not asupplemental enhancement information flag within a supplementalenhancement information of the output video data stream, or a pictureparameter set flag within a picture parameter set of the output videodata stream, or a sequence parameter set flag within a sequenceparameter set of the output video data stream, or an external meansflag, wherein a value of the external means flag may, e.g., be set by anexternal unit being external to an apparatus 200.

Furthermore, a video encoder 100 is provided. The video encoder 100 may,e.g., be configured to encode a video into a video data stream.Moreover, the video encoder 100 may, e.g., be configured to generate thevideo data stream such that the video data stream comprises anindication that indicates whether a picture of the video preceding adependent random access picture shall be output or not.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise supplemental enhancement informationcomprising the indication that may, e.g., indicate whether the pictureof the video preceding the dependent random access picture shall beoutput or not.

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The video encoder 100 may, e.g., be configured to generate thevideo data stream such that the video data stream may, e.g., comprise aflag (e.g., a ph_pic_output_flag) having a predefined value (e.g., 0) ina picture header of the independent random access picture, such that thepredefined value (e.g., 0) of the flag (e.g., a ph_pic_output_flag) may,e.g., indicate for the independent random access picture directlyprecedes said dependent random access picture within the video datastream, and that said independent random access picture shall not beoutput.

According to an embodiment, the flag may, e.g., be a first flag, whereinthe video encoder 100 may, e.g., be configured to generate the videodata stream such that the video data stream may, e.g., comprise afurther flag in a picture parameter set of the video data stream,wherein the further flag may, e.g., indicate whether or not the firstflag (e.g., a ph_pic_output_flag) exists in the picture header of theindependent random access picture.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that the video data stream may,e.g., comprise as the indication that may, e.g., indicate whether thepicture of the video preceding the dependent random access picture shallbe output or not a supplemental enhancement information flag within asupplemental enhancement information of the output video data stream, ora picture parameter set flag within a picture parameter set of theoutput video data stream, or a sequence parameter set flag within asequence parameter set of the output video data stream, or an externalmeans flag, wherein a value of the external means flag may, e.g., be setby an external unit being external to an apparatus 200.

Moreover, a video decoder 300 for receiving a video data stream having avideo stored therein is provided. The video decoder 300 is configured todecode the video from the video data stream. The video decoder 300 isconfigured to decode the video depending on an indication indicatingwhether a picture of the video preceding a dependent random accesspicture shall be output or not.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to decode the video depending on a first variable (e.g., aNoOutputBeforeDrapFlag) indicating whether the picture of the video thatprecedes the dependent random access picture shall be output or not.

In an embodiment, the video data stream may, e.g., comprise theindication that may, e.g., indicate whether the picture of the videopreceding the dependent random access picture shall be output or not.The video decoder 300 may, e.g., be configured to decode the videodepending on the indication within the video data stream.

According to an embodiment, the video data stream may, e.g., comprisesupplemental enhancement information comprising the indication that may,e.g., indicate whether the picture of the video preceding the dependentrandom access picture shall be output or not.

The video decoder 300 may, e.g., be configured to decode the videodepending on the supplemental enhancement information.

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The video data stream may, e.g., comprise a flag (e.g., aph_pic_output_flag) having a predefined value (e.g., 0) in a pictureheader of the independent random access picture, such that thepredefined value (e.g., 0) of the flag (e.g., a ph_pic_output_flag) may,e.g., indicate for the independent random access picture directlyprecedes said dependent random access picture within the video datastream, and that said independent random access picture shall not beoutput. The video decoder 300 may, e.g., be configured to decode thevideo depending on the flag.

According to an embodiment, the flag may, e.g., be a first flag, whereinthe video data stream may, e.g., comprise a further flag in a pictureparameter set of the video data stream, wherein the further flag may,e.g., indicate whether or not the first flag (e.g., aph_pic_output_flag) exists in the picture header of the independentrandom access picture. The video decoder 300 may, e.g., be configured todecode the video depending on the further flag.

In an embodiment, the video data stream may, e.g., comprise as theindication that may, e.g., indicate whether the picture of the videopreceding the dependent random access picture shall be output or not asupplemental enhancement information flag within a supplementalenhancement information of the output video data stream, or a pictureparameter set flag within a picture parameter set of the output videodata stream, or a sequence parameter set flag within a sequenceparameter set of the output video data stream, or an external meansflag, wherein a value of the external means flag may, e.g., be set by anexternal unit being external to an apparatus 200. The video decoder 300may, e.g., be configured to decode the video depending on the indicationwithin the video data stream.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to reconstruct the video from the video data stream. Thevideo decoder 300 may, e.g., be configured to output or to not outputthe picture of the video that precedes the dependent random accesspicture depending on the first variable (e.g., aNoOutputBeforeDrapFlag).

In an embodiment, the video decoder 300 may, e.g., be configured todetermine a value of a second variable (e.g., a PictureOutputFlag) forthe picture of the video that precedes the dependent random accesspicture depending on the first variable (e.g., aNoOutputBeforeDrapFlag), wherein the second variable (e.g., aPictureOutputFlag) may, e.g., indicate for said picture whether saidpicture shall be output or not, and wherein the apparatus 200 may, e.g.,be configured to output or to not output said picture depending on thesecond variable (e.g., a PictureOutputFlag).

According to an embodiment, the picture of the video that precedes thedependent random access picture may, e.g., be an independent randomaccess picture. The video decoder 300 may, e.g., be configured to decodethe video depending on the first variable (e.g., aNoOutputBeforeDrapFlag) which may, e.g., indicate that the independentrandom access picture shall not be output.

In an embodiment, the picture of the video that precedes the dependentrandom access picture may, e.g., be an independent random accesspicture. The video decoder 300 may, e.g., be configured to decode thevideo depending on the first variable (e.g., a NoOutputBeforeDrapFlag)which may, e.g., indicate that the independent random access pictureshall be output.

Furthermore, a system is provided. The system comprises an apparatus 200as described above and a video decoder 300 as described above. The videodecoder 300 is configured to receive the output video data stream of theapparatus 200. Moreover, the video decoder 300 is configured to decodethe video from the output video data stream of the apparatus 200.

According to an embodiment, the system may, e.g., further comprise avideo encoder 100. The apparatus 200 may, e.g., be configured to receivethe video data stream from the video encoder 100 as the input video datastream.

In particular, the first aspect of the invention relates to CVS start atDRAP and to omit IDR output in decoding and conformance testing.

When a bitstream comprises pictures marked as DRAP (i.e. only using theprevious IRAP as reference for the DRAP and from there on in thebitstream) it is possible to utilize these DRAP pictures for randomaccess functionality at lower rate overhead. However, when using sometarget DRAP for random accessing a stream, it is undesirable to displayany initial picture before the target DRAP (i.e. the associated IRAP oftarget DRAP) at the decoder output as the temporal distance betweenthese pictures would lead to a shacky/jittery video playback when playedback at the rate of the original video until the video is played back ina smooth way from the target DRAP on.

Therefore, it is desirable to omit the output of the pictures precedingthe DRAP picture. This aspect of the invention presents means to controlthe decoder accordingly.

In one embodiment, an external means to set the PicOutputFlag variableof an IRAP picture is made available for implementations to use asfollows:

-   -   If some external means not specified in this Specification is        available to set the variable NoOutputBeforeDrapFlag for the        picture to a value, NoOutputBeforeDrapFlag for the picture is        set equal to the value provided by the external means.

[ . . . ]

-   -   The variable PictureOutputFlag of the current picture is derived        as follows:        -   If sps_video_parameter_set_id is greater than 0 and the            current layer is not an output layer (i.e., nuh_layer_id is            not equal to OutputLayerIdInOIs[TargetOIsIdx][i] for any            value of i in the range of 0 to            NumOutputLayersInOIs[TargetOIsIdx]−1, inclusive), or one of            the following conditions is true, PictureOutputFlag is set            equal to 0:—            -   The current picture is a RASL picture and                NoOutputBeforeRecoveryFlag of the associated IRAP                picture is equal to 1.            -   The current picture is a GDR picture with                NoOutputBeforeRecoveryFlag equal to 1 or is a recovering                picture of a GDR picture with NoOutputBeforeRecoveryFlag                equal to 1.            -   The current picture is a IRAP picture with                NoOutputBeforeDrapFlag equal to 1.        -   Otherwise, PictureOutputFlag is set equal to            ph_pic_output_flag.

In another embodiment, the NoOutputBeforeDrapFlag is set by externalmeans only for the first IRAP picture in a CVS, and set to 0 otherwise.

-   -   If some external means not specified in this Specification is        available to set the variable NoOutputBeforeDrapFlag for the        picture to a value, NoOutputBeforeDrapFlag for the first picture        in the CVS is set equal to the value provided by the external        means. Otherwise, NoOutputBeforeDrapFlag is set to 0.

The above mentioned flag NoOutputBeforeDrapFlag could also be associatedwith the usage of alternative HRD timings conveyed in the bitstream forthe case of removal of pictures between the IRAP picture and the DRAPpicture, e.g. a the flag UseAltCpbParamsFlag in the VVC specification.

In an alternative embodiment, it is a constraint that IRAP pictures thatdirectly precede DRAP pictures without non-DRAP pictures in betweenshall have a value of 0 in the output flag ph_pic_output_flag in thepicture header. In this case, whenever an extractor or player uses aDRAP for random accessing, i.e. it removes intermediate pictures betweenIRAP and DRAP from the bitstream, it is also required to verify oradjust that the respective output flag is set to 0 and output of theIRAP is omitted.

For this operation to be simple the original bitstream needs to beprepared correspondingly. More concretely, pps_output_flag_present_flag,which determines the presence of the flag ph_pic_output_flag in thepicture header shall be equal to 1 so that the picture header can beeasily change and it is not required to change also parameter sets. Thatis:

It is a requirement of bitstream conformance that the value ofpps_output_flag_present_flag shall be equal to 1 if the PPS is referredto by a picture within a CVSS AU that has associated DRAP AUs.

In addition to the options listed above, in another embodiment, it isindicated in a parameter set PPS or an SPS whether the first AU in thebitstream, i.e. a CRA or IDR that constitutes a CLVS start, is to beoutput or not after decoding. Thus, the system integration is simpler,as only a parameter set needs to be adjusted instead of requiringcomparatively low-level syntax such as a PHs also to be changed, e.g.when parsing a file in file format ISOBMFF.

An example is shown in the following:

Descriptor seq_parameter_set_rbsp( ) {  [...]  if(sps_conformance_window_flag ) {   sps_conf_win_left_offset ue(v)  sps_conf_win_right_offset ue(v)   sps_conf_win_top_offset ue(v)  sps_conf_win_bottom_offset ue(v)  }  sps_pic_in_cvss_au_no_output_flagu(1)  sps_log2_ctu_size_minus5 u(2)   [...] }

sps_pic_in_cvss_au_no_output_flag equal to 1 specifies that a picture ina CVSS AU referring to the SPS is not output.sps_pic_in_cvss_au_no_output_flag equal to 0 specifies that a picture ina CVSS AU referring to the SPS may or may not be output.

It is a requirement of bitstream conformance that the value ofsps_pic_in_cvss_au_no_output_flag shall be the same for any SPS referredto by any output layer in an OLS.

In 8.1.2

-   -   The variable PictureOutputFlag of the current picture is derived        as follows:        -   If sps_video_parameter_set_id is greater than 0 and the            current layer is not an output layer (i.e., nuh_layer_id is            not equal to OutputLayerIdInOIs[TargetOIsIdx][i] for any            value of i in the range of 0 to            NumOutputLayersInOIs[TargetOIsIdx]−1, inclusive), or one of            the following conditions is true, PictureOutputFlag is set            equal to 0:            -   The current picture is a RASL picture and                NoOutputBeforeRecoveryFlag of the associated IRAP                picture is equal to 1.            -   The current picture is a GDR picture with                NoOutputBeforeRecoveryFlag equal to 1 or is a recovering                picture of a GDR picture with NoOutputBeforeRecoveryFlag                equal to 1.        -   Otherwise, if the current AU is a CVSS AU and            sps_pic_in_cvss_au_no_output_flag equal to 1            PictureOutputFlag is set equal to 0.        -   Otherwise, PictureOutputFlag is set equal to            ph_pic_output_flag.            -   NOTE—In an implementation, the decoder could output a                picture not belonging to an output layer. For example,                when there is only one output layer while in an AU the                picture of the output layer is not available, e.g., due                to a loss or layer down-switching, the decoder could set                PictureOutputFlag set equal to 1 for the picture that                has the highest value of nuh_layer_id among all pictures                of the AU available to the decoder and having                ph_pic_output_flag equal to 1, and set PictureOutputFlag                equal to 0 for all other pictures of the AU available to                the decoder.

In another embodiment, for example, a requirement may, e.g., be definedas follows:

It is a requirement of bitstream conformance that the value ofph_pic_output_flag shall be equal to 0 if the a picture belongs to aIRAP AU and the IRAP AU is directly preceding a DRAP AU.

In the following, the second aspect of the invention is now described indetail.

In accordance with the second aspect of the invention, an apparatus 200for receiving one or more input video data streams is provided. Each ofthe one or more input video data streams has an input video encodedthereinto. The apparatus 200 is configured to generate an output videodata stream from the one or more input video data streams, the outputvideo data stream encoding an output video, wherein the apparatus isconfigured to generate the output video data stream such that the outputvideo is the input video being encoded within one of the one or moreinput video data streams, or such that the output video depends on theinput video of at least one of the one or more input video data streams.Moreover, the apparatus 200 is configured to determine an access unitremoval time of a current picture of a plurality of pictures of theoutput video from a coded picture buffer. The apparatus 200 isconfigured to determine whether or not to use coded picture buffer delayoffset information for determining the access unit removal time of thecurrent picture from the coded picture buffer.

According to an embodiment, the apparatus 200 may, e.g., be configuredto drop a group of one or more pictures of the input video of a firstvideo data stream of the one or more input video data streams togenerate the output video data stream. The apparatus 200 may, e.g., beconfigured to determine an access unit removal time for at least one ofthe plurality of pictures of the output video from the coded picturebuffer depending on the coded picture buffer delay offset information.

In an embodiment, the first video received by the apparatus 200 may,e.g., be a preprocessed video which results from an original video fromwhich a group of one or more pictures has been dropped to generate thepreprocessed video. The apparatus 200 may, e.g., be configured todetermine an access unit removal time for at least one of the pluralityof pictures of the output video from the coded picture buffer dependingon the coded picture buffer delay offset information.

According to an embodiment, the buffer delay offset information dependson a number of pictures of the input video that have been dropped.

In an embodiment, the one or more input video data streams are two ormore input video data streams. The apparatus 200 may, e.g., beconfigured to splice the processed video and the input video of a secondvideo data stream of the two or more input video data streams to obtainthe output video, and may, e.g., be configured to encode the outputvideo into the output video data stream.

According to an embodiment, the apparatus 200 may, e.g., be configuredto determine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on a location of the current picture within the outputvideo. Or, the apparatus 200 may, e.g., be configured to determinewhether or not to set a coded picture buffer delay offset value of thecoded picture buffer delay offset information to 0 for determining theaccess unit removal time of the current picture depending on thelocation of the current picture within the output video.

In an embodiment, the apparatus 200 may, e.g., be configured todetermine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on a position of a previous non-discardable picturewhich precedes the current picture within the output video.

According to an embodiment, the apparatus 200 may, e.g., be configuredto determine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on whether or not the previous non-discardable picturewhich precedes the current picture within the output video may, e.g., bea first picture in a previous buffering period.

In an embodiment, the apparatus 200 may, e.g., be configured todetermine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on a concatenation flag, the current picture being afirst picture of the input video of the second video data stream.

According to an embodiment, the apparatus 200 may, e.g., be configuredto determine the access unit removal time of the current picturedepending on a removal time of a preceding picture.

In an embodiment, the apparatus 200 may, e.g., be configured todetermine the access unit removal time of the current picture dependingon initial coded picture buffer removal delay information.

According to an embodiment, the apparatus 200 may, e.g., be configuredto update the initial coded picture buffer removal delay informationdepending on a clock tick to obtain temporary coded picture bufferremoval delay information to determine the access unit removal time ofthe current picture.

According to an embodiment, if the concatenation flag is set to a firstvalue, then the apparatus 200 is configured to use the coded picturebuffer delay offset information to determine one or more removal times.If the concatenation flag is set to a second value being different fromthe first value then the apparatus 200 is configured to not use thecoded picture buffer delay offset information to determine the one ormore removal times.

In an embodiment, the apparatus 200 may, e.g., be configured to signalto a video decoder 300, whether or not to use coded picture buffer delayoffset information for determining the access unit removal time of thecurrent picture from the coded picture buffer.

According to an embodiment, the current picture may, e.g., be located ata splicing point of the output video, where two input videos have beenspliced.

Furthermore, a video data stream is provided. The video data stream hasa video encoded thereinto. The video data stream comprises coded picturebuffer delay offset information.

According to an embodiment, the video data stream may, e.g., comprise aconcatenation flag.

In an embodiment, the video data stream may, e.g., comprise initialcoded picture buffer removal delay information.

According to an embodiment, if the concatenation flag is set to a firstvalue (e.g., 0) then the concatenation flag indicates that the codedpicture buffer delay offset information needs to be used to determineone or more (picture or access unit) removal times, e.g., when it isknown that some pictures (e.g., RASL pictures) have been dropped. If theconcatenation flag is set to a second value being different from thefirst value (e.g., 1) then the concatenation flag indicates that theindicated offset is not used to determine the one or more (picture oraccess unit) removal times, e.g., irrespective of an offset signalingand, e.g., irrespective of whether RASL pictures have been dropped. Ifpictures are not dropped, then, e.g., the offset is not to be used.

Moreover, a video encoder 100, is provided. The video encoder 100 isconfigured to encode a video into a video data stream. The video encoder100 is configured to generate the video data stream such that the videodata stream comprises coded picture buffer delay offset information.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise a concatenation flag.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that the video data stream may,e.g., comprise coded picture buffer delay offset information.

Furthermore, a video decoder 300 for receiving a video data stream avideo stored therein is provided. The video decoder 300 is configured todecode the video from the video data stream. Moreover, the video decoder300 is configured to decode the video depending on an access unitremoval time of a current picture of a plurality of pictures of thevideo from a coded picture buffer. The video decoder 300 is configuredto decode the video depending on an indication indicating whether or notto use coded picture buffer delay offset information for determining theaccess unit removal time of the current picture from the coded picturebuffer.

According to an embodiment, the access unit removal time for at leastone of the plurality of pictures of the video from the coded picturebuffer depends on the coded picture buffer delay offset information.

In an embodiment, the video decoder 300 is configured to decode thevideo depending on whether or not to use coded picture buffer delayoffset information for determining the access unit removal time of thecurrent picture depending on a location of the current picture withinthe video.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to decode the video depending on whether or not a codedpicture buffer delay offset value of the coded picture buffer delayoffset information may, e.g., be set to 0.

In an embodiment, the video decoder 300 may, e.g., be configured todetermine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on a position of a previous non-discardable picturewhich precedes the current picture within the video.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to determine whether or not to use coded picture buffer delayoffset information for determining the access unit removal time of thecurrent picture depending on whether or not the previous non-discardablepicture which precedes the current picture within the video may, e.g.,be a first picture in a previous buffering period.

In an embodiment, the video decoder 300 may, e.g., be configured todetermine whether or not to use coded picture buffer delay offsetinformation for determining the access unit removal time of the currentpicture depending on a concatenation flag, the current picture being afirst picture of the input video of the second video data stream.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to determine the access unit removal time of the currentpicture depending on a removal time of a preceding picture.

In an embodiment, the video decoder 300 may, e.g., be configured todetermine the access unit removal time of the current picture dependingon initial coded picture buffer removal delay information.

According to an embodiment, the video decoder 300 may, e.g., beconfigured to update the initial coded picture buffer removal delayinformation depending on a clock tick to obtain temporary coded picturebuffer removal delay information to determine the access unit removaltime of the current picture.

According to an embodiment, if the concatenation flag is set to a firstvalue, then the video decoder 300 is configured to use the coded picturebuffer delay offset information to determine one or more removal times.If the concatenation flag is set to a second value being different fromthe first value then the video decoder 300 is configured to not use thecoded picture buffer delay offset information to determine the one ormore removal times.

Furthermore, a system is provided. The system comprises an apparatus 200as described above and a video decoder 300 as described above. The videodecoder 300 is configured to receive the output video data stream of theapparatus 200. Moreover, the video decoder 300 is configured to decodethe video from the output video data stream of the apparatus 200.

According to an embodiment, the system may, e.g., further comprise avideo encoder 100. The apparatus 200 may, e.g., be configured to receivethe video data stream from the video encoder 100 as the input video datastream.

In particular, the second aspect of the invention relates to thatprevNonDiscardable in case of alternative timing may (when it is not aBP start) already include the alternative offset (CpbDelayOffset), sofor the AU with concatenation_flag==1, CpbDelayOffset should betemporally set to zero.

When splicing of two bitstreams happens the derivation of the removaltime of an AU from the CPB is done differently than for non-splicedbitstreams. At the splicing point a Buffering Period SEI message (BP SEImessage; SEI=supplemental enhancement information) comprises aconcatenationFlag being equal to 1. Then the decoder needs to checkamong 2 values and take the bigger one of both:

-   -   previous Non Discardable Pic (prevNonDiscardablePic) removal        time plus a delta signalled in the BP SEI message        (auCpbRemovalDelayDeltaMinus1+1), or    -   preceding Pic removal time plus InitCpbRemovalDelay

However, when the previous Pic with a BP SEI message was an AU for whichalternative timings have been used (i.e. a second timing informationused when the RASL picture or pictures up to a DRAP have been dropped)for derivation of the removal times, an offset is used (CpbDelayOffset)to compute each removal time that computed as a delta to the previouspic with a buffering period, i.e.AuNominalRemovalTime[firstPicInPrevBuffPeriod] plusAuCpbRemovalDelayVal−CpbDelayOffset, as illustrated in FIG. 4 .

FIG. 4 illustrates an original bitstream (top of FIG. 4 ), and abitstream after dropping pictures (bottom of FIG. 4 ): An offset isincorporated into a calculation of the removal delay after dropping AUs(lines 1, 2 and 3 in the original bitstream).

The offset is added since the removal time is computed using a delta tothe removal time of the picture referred to as firstPicInPrevBuffPeriod,after which some AUs have been dropped and therefore a CpbDelayOffset isnecessary to account (compensate) for the AU dropping.

FIG. 5 illustrates the splicing of two bitstreams (at a differentposition), a first bitstream (in FIG. 5 middle left) and a secondbitstream (in FIG. 5 , middle, right), after pictures were dropped fromthe original first bitstream (in FIG. 5 , middle, left).

The example for using the preceding Pic removal time as anchor insteadof the previous non-discardable picture is similar and would not requirethe “−3” correction factor (CpbDelayOffset) neither.

However, in the splicing case as illustrated in FIG. 5 , note that it isnot necessarily the case that the two derivations use the removal timeof the AU associated with a BP SEI message (firstPicInPrevBuffPeriod).As discussed, for the splicing case a delta added to either theprevNonDiscardablePic or just the preceding Pic. This means that whenthe prevNonDiscardablePic is not the firstPicInPrevBuffPeriod, theCpbDelayOffset cannot be used to derive the removal time from the CPB ofthe current AU, as the removal time of prevNonDiscardablePic alreadyaccounts for AU dropping and no AUs are dropped betweenprevNonDiscardablePic and the AU for which the removal time is computed.Now imagine that the preceding Pic removal time is used instead, as forthe case that the current AU (i.e. the splicing point with a new BP SEImessage) has an InitialCpbRemovalDelay that forces the removal time ofthe current AU to come after its desired removal time, which would haveachieved an equidistant removal time (when the prevNonDiscardablePic isused instead). In such a case, the removal time of the current AU cannotbe smaller than the time computed by using the preceding Pic removaltime plus InitCpbRemovalDelay as this could lead to buffer underruns(AUs not in the buffer before they need to be removed). Therefore, aspart of the invention, for this case the CpbDelayOffset is not used forthe computation or considered to be equal to 0.

Summarizing the embodiment herein is to use a CpbDelayOffset for thecomputation of AU removal times when RASL AUs are dropped from abitstream or AUs in between an IRAP and DRAP AUs are dropped dependingon a check. The check to determine whether CpbDelayOffset is not used orconsidered to be equal to 0 being one of the following:

-   -   prevNonDiscardablePic is not the firstPicInPrevBuffPeriod    -   preceding Pic removal time plus InitCpbRemovalDelay is used for        the computation of the removal of the current AU

The implementation in the specification could be as follows:

-   -   When AU n is the first AU of a BP that does not initialize the        HRD, the following applies:    -   The nominal removal time of the AU n from the CPB is specified        by:

if(\!concatenationFlag){baseTime = AuNominalRemovalTime[firstPicInPrevBuffPeriod]tmpCpbRemovalDelay = AuCpbRemovalDelayValtmpCpbDelayOffset = CpbDelayOffset}else{baseTime1 = AuNominalRemovalTime[prevNonDiscardablePic]tmpCpbRemovalDelay1 = (auCpbRemovalDelayDeltaMinus1 + 1)baseTim2 = AuNominalRemovalTime[n − 1]$\begin{matrix}{\left. {{{tmpCpbRemovalDelay}2} = {{Ceil}{{\left( {\left( {{{{{InitCpbRemovalDelay}\lbrack{Htid}\rbrack}\lbrack{ScIdx}\rbrack} \div 90000} + {{AuFinalArrivalTime}{{\left\lbrack {n - 1} \right\rbrack - {{AuNominalRemovalTime}\left\lbrack {n - 1} \right\rbrack}}}}} \right) \div {ClockTick}} \right){if}\left( {{{{baseTime}1} + {{ClockTick}*{tmpCpbRemovalDelay}1}} < {{{baseTime}2} + {{ClockTick}*{tmpCpbRemovalDelay}2}}} \right)\left\{ {{baseTime} = {{{baseTime}2{tmpCpbRemovalDelay}} = {{{tmpCpbRemovalDelay}2{tmpCpbDelayOffset}} = 0}}} \right\}{else}\left\{ {{baseTime} = {{{baseTime}1{tmpCpbRemovalDelay}} = {{{tmpCpbRemovalDelay}1{tmpCpbDelayOffset}} = \left( {\left( {{prevNonDiscardablePic}=={firstPicInPrevBuffPeriod}} \right)?{{CpbDelayOffset}:0}} \right)}}} \right\}}}}} \right\}{{{AuNominalRemovalTime}\lbrack n\rbrack} = {{baseTime} + \left( {{{ClockTick}*{tmpCpbRemovalDelay}} - {tmpCpbDelayOffset}} \right)}}} & \left( {C\text{.10}} \right)\end{matrix}$

Alternatively, in another embodiment illustrated in FIG. 6 , theCpbDelayOffset for the computation of AU removal times when RASL AUs aredropped from a bitstream or AUs in between an IRAP and DRAP AUs aredropped depending on a different check that comprises checking theconcatenationFlag.

In that case the delta in the bitstream when concatenationFlag is set to1 needs to match the proper value as if the CpbDelayOffset was accountedfor (as evident when comparing FIGS. 5 and 6 ), as for that figureCpbDelayOffset is not applied or considered to be 0.

The implementation in the specification could be as follows:

-   -   When AU n is the first AU of a BP that does not initialize the        HRD, the following applies:    -   The nominal removal time of the AU n from the CPB is specified        by:

if(\!concatenationFlag){baseTime = AuNominalRemovalTime[firstPicInPrevBuffPeriod]tmpCpbRemovalDelay = AuCpbRemovalDelayValtmpCpbDelayOffset = CpbDelayOffset}else{baseTime1 = AuNominalRemovalTime[prevNonDiscardablePic]tmpCpbRemovalDelay1 = (auCpbRemovalDelayDeltaMinus1 + 1)baseTim2 = AuNominalRemovalTime[n − 1]$\begin{matrix}{\left. {{{tmpCpbRemovalDelay}2} = {{Ceil}{{{\left( {\left( {{{{{InitCpbRemovalDelay}\lbrack{Htid}\rbrack}\lbrack{ScIdx}\rbrack} \div 90000} + {{AuFinalArrivalTime}{{\left\lbrack {n - 1} \right\rbrack - {{AuNominalRemovalTime}\left\lbrack {n - 1} \right\rbrack}}}}} \right) \div {ClockTick}} \right){if}\left( {{{{baseTime}1} + {{ClockTick}*{tmpCpbRemovalDelay}1}} < {{{baseTime}2} + {{ClockTick}*{tmpCpbRemovalDelay}2}}} \right)\left\{ {{baseTime} = {{{baseTime}2{tmpCpbRemovalDelay}} = {{tmpCpbRemovalDelay}2}}} \right\}{else}\left\{ {{baseTime} = {{{baseTime}1{tmpCpbRemovalDelay}} = {{tmpCpbRemovalDelay}1}}} \right\}{tmpCpbDelayOffset}} = 0}}}} \right\}{{{AuNominalRemovalTime}\lbrack n\rbrack} = {{baseTime} + \left( {{{ClockTick}*{tmpCpbRemovalDelay}} - {tmpCpbDelayOffset}} \right)}}} & \left( {C\text{.10}} \right)\end{matrix}$

In the following, the third aspect of the invention is now described indetail.

In accordance with the third aspect of the invention, a video datastream is provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises an initial coded picturebuffer removal delay. Furthermore, the video data stream comprises aninitial coded picture buffer removal offset. Moreover, the video datastream comprises information that indicates whether or not a sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset is defined to be constant across two or morebuffering periods.

According to an embodiment, the initial coded picture buffer removaldelay may, e.g., indicate a time that needs to pass for a first accessunit of a picture of the video data stream that initializes a videodecoder 300 before sending the first access unit to the video decoder300.

In an embodiment, the video data stream may, e.g., comprise a singleindication that may, e.g., indicate whether or not the sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset may, e.g., be defined to be constant across thetwo or more buffering periods.

According to an embodiment, the video data stream may, e.g., comprise aconcatenation flag as the single indication, that may, e.g., indicatewhether or not the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset may, e.g., bedefined to be constant across the two or more buffering periods. If theconcatenation flag is equal to a first value, the sum of the initialcoded picture buffer removal delay and the initial coded picture bufferremoval offset is constant across the two or more buffering periods. Ifthe concatenation flag is different from the first value, theconcatenation flag does not define whether or not the sum of the initialcoded picture buffer removal delay and the initial coded picture bufferremoval offset is constant across the two or more buffering periods.

In an embodiment, if the single indication does not indicate that thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant across thetwo or more buffering periods, the video data stream may, e.g., comprisecontinuously updated information on the initial coded picture bufferremoval delay information and continuously updated information on theinitial coded picture buffer removal offset information.

According to an embodiment, if the video data stream comprises theinformation that indicates that the sum of the initial coded picturebuffer removal delay and the initial coded picture buffer removal offsetis defined to be constant across the two or more buffering periods, thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset may, e.g., be defined to be constantstarting from a current position within the video data stream.

Furthermore, a video encoder 100 is provided. The video encoder 100 isconfigured to encode a video into a video data stream. Moreover, thevideo encoder 100 is configured to generate the video data stream suchthat the video data stream comprises an initial coded picture bufferremoval delay. Furthermore, the video encoder 100 is configured togenerate the video data stream such that the video data stream comprisesan initial coded picture buffer removal offset. Moreover, the videoencoder 100 is configured to generate the video data stream such thatthe video data stream comprises information that indicates whether ornot a sum of the initial coded picture buffer removal delay and theinitial coded picture buffer removal offset is defined to be constantacross two or more buffering periods.

According to an embodiment, the initial coded picture buffer removaldelay may, e.g., indicate a time that needs to pass for a first accessunit of a picture of the video data stream that initializes a videodecoder 300 before sending the first access unit to the video decoder300.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that the video data stream may,e.g., comprise a single indication that may, e.g., indicate whether ornot the sum of the initial coded picture buffer removal delay and theinitial coded picture buffer removal offset may, e.g., be defined to beconstant across the two or more buffering periods.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise a concatenation flag as the singleindication, that may, e.g., indicate whether or not the sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset may, e.g., be defined to be constant across thetwo or more buffering periods. If the concatenation flag is equal to afirst value, the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset is constant acrossthe two or more buffering periods. If the concatenation flag isdifferent from the first value, the concatenation flag does not definewhether or not the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset is constant acrossthe two or more buffering periods.

In an embodiment, if the single indication does not indicate that thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant across thetwo or more buffering periods, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise continuously updated information on theinitial coded picture buffer removal delay information and continuouslyupdated information on the initial coded picture buffer removal offsetinformation.

According to an embodiment, if the video data stream comprises theinformation that may, e.g., indicate that the sum of the initial codedpicture buffer removal delay and the initial coded picture bufferremoval offset is defined to be constant across the two or morebuffering periods, the sum of the initial coded picture buffer removaldelay and the initial coded picture buffer removal offset is defined tobe constant starting from a current position within the video datastream.

Moreover, an apparatus 200 for receiving two input video data streams,being a first input video data stream and a second input video datastream, is provided. Each of the two input video data streams has aninput video encoded thereinto. The apparatus 200 is configured togenerate an output video data stream from the two input video datastreams, the output video data stream encoding an output video, whereinthe apparatus is configured to generate an output video data stream byconcatenating the first input video data stream and the second inputvideo data stream. Moreover, the apparatus 200 is configured to generatethe output video data stream such that the output video data streamcomprises an initial coded picture buffer removal delay. Furthermore,the apparatus 200 is configured to generate the output video data streamsuch that the output video data stream comprises an initial codedpicture buffer removal offset. Moreover, the apparatus 200 is configuredto generate the output video data stream such that the output video datastream comprises information that indicates whether or not a sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset is defined to be constant across two or morebuffering periods.

According to an embodiment, the initial coded picture buffer removaldelay may, e.g., indicate a time that needs to pass for a first accessunit of a picture of the output video data stream that initializes avideo decoder 300 before sending the first access unit to the videodecoder 300.

In an embodiment, the apparatus 200 may, e.g., be configured to generatethe output video data stream such that the output video data stream may,e.g., comprise a single indication that may, e.g., indicate whether ornot the sum of the initial coded picture buffer removal delay and theinitial coded picture buffer removal offset may, e.g., be defined to beconstant across the two or more buffering periods.

According to an embodiment, the apparatus 200 may, e.g., be configuredto generate the output video data stream such that the output video datastream may, e.g., comprise a concatenation flag as the singleindication, that may, e.g., indicate whether or not the sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset may, e.g., be defined to be constant across thetwo or more buffering periods. If the concatenation flag is equal to afirst value, the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset is constant acrossthe two or more buffering periods. If the concatenation flag isdifferent from the first value, the concatenation flag does not definewhether or not the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset is constant acrossthe two or more buffering periods.

In an embodiment, if the single indication does not indicate that thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant across thetwo or more buffering periods, the apparatus 200 is configured togenerate the output video data stream such that the output video datastream comprises continuously updated information on the initial codedpicture buffer removal delay information and continuously updatedinformation on the initial coded picture buffer removal offsetinformation.

According to an embodiment, if the video data stream comprises theinformation that indicates that the sum of the initial coded picturebuffer removal delay and the initial coded picture buffer removal offsetis defined to be constant across the two or more buffering periods, thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant startingfrom a current position within the video data stream.

Furthermore, a video decoder 300 for receiving a video data stream avideo stored therein is provided. The video decoder 300 is configured todecode the video from the video data stream. Moreover, the video datastream comprises an initial coded picture buffer removal delay.Furthermore, the video data stream comprises an initial coded picturebuffer removal offset. Moreover, the video data stream comprisesinformation that indicates whether or not a sum of the initial codedpicture buffer removal delay and the initial coded picture bufferremoval offset is defined to be constant across two or more bufferingperiods. Furthermore, the video decoder 300 is configured to decode thevideo depending on the information that indicates whether or not the sumof the initial coded picture buffer removal delay and the initial codedpicture buffer removal offset is defined to be constant across the twoor more buffering periods.

According to an embodiment, the initial coded picture buffer removaldelay may, e.g., indicate a time that needs to pass for a first accessunit of a picture of the output video data stream that initializes thevideo decoder 300 before sending the first access unit to the videodecoder 300.

In an embodiment, the video data stream may, e.g., comprise a singleindication that may, e.g., indicate whether or not the sum of theinitial coded picture buffer removal delay and the initial coded picturebuffer removal offset may, e.g., be defined to be constant across thetwo or more buffering periods. The video decoder 300 may, e.g., beconfigured to decode the video depending on the single indication.

According to an embodiment, the video data stream may, e.g., comprise aconcatenation flag as the single indication, that may, e.g., indicatewhether or not the sum of the initial coded picture buffer removal delayand the initial coded picture buffer removal offset may, e.g., bedefined to be constant across the two or more buffering periods. If theconcatenation flag is equal to a first value, the sum of the initialcoded picture buffer removal delay and the initial coded picture bufferremoval offset is constant across the two or more buffering periods. Ifthe concatenation flag is different from the first value, theconcatenation flag does not define whether or not the sum of the initialcoded picture buffer removal delay and the initial coded picture bufferremoval offset is constant across the two or more buffering periods. Thevideo decoder 300 is configured to decode the video depending on theconcatenation flag.

In an embodiment, if the single indication does not indicate that thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant across thetwo or more buffering periods, the video data stream comprisescontinuously updated information on the initial coded picture bufferremoval delay information and continuously updated information on theinitial coded picture buffer removal offset information. The videodecoder 300 is configured to decode the video depending on thecontinuously updated information on the initial coded picture bufferremoval delay information and on the continuously updated information onthe initial coded picture buffer removal offset information.

According to an embodiment, if the video data stream comprises theinformation that indicates that the sum of the initial coded picturebuffer removal delay and the initial coded picture buffer removal offsetis defined to be constant across the two or more buffering periods, thesum of the initial coded picture buffer removal delay and the initialcoded picture buffer removal offset is defined to be constant startingfrom a current position within the video data stream.

Moreover, a system is provided. The system comprises an apparatus 200 asdescribed above and a video decoder 300 as described above. The videodecoder 300 is configured to receive the output video data stream of theapparatus 200. Moreover, the video decoder 300 is configured to decodethe video from the output video data stream of the apparatus 200.

According to an embodiment, the system may, e.g., further comprise avideo encoder 100. The apparatus 200 according to one of claims 221 to226 may, e.g., be configured to receive the video data stream from thevideo encoder 100 according to one of claims 211 to 216 as the inputvideo data stream.

In particular, the third aspect of the invention relates to splicing, toan Initial Cpb Removal Delay and to an Initial Cpb Removal Offset

Currently the specification indicates that the sum of Initial CpbRemoval Delay and Initial Cpb Removal Offset are constant within a CVS.The same constraint is expressed for the alternative timings. TheInitial Cpb Removal Delay indicates the time that needs to pass for thefirst AU in the bitstream that initializes the decoder before sendingthe first AU for decoding. The Initial Cpb Removal Offset is a propertyof the bitstream that means that the AUs earliest arrival time in thedecoder is not necessarily equidistant with respect to the time 0 atwhich the first AU arrive the decoder. It helps determining when thefirst bit of an AU can earliest reach the decoder.

The current constraint in the WC draft specification indicates that thesum of these two values needs to be constant within a CVS:

-   -   Over the entire CVS, for each value pair of i and j, the sum of        nal_initial_cpb_removal_delay[i][j] and        nal_initial_cpb_removal_offset[i][j] shall be constant, and the        sum of nal_initial_alt_cpb_removal_delay[i][j] and        nal_initial_alt_cpb_removal_offset[i][j] shall be constant.

The problem appears when editing or splicing bitstreams to form a newjoint bitstream. It is desirable also to be able to indicate whetherthis property is fulfilled across the CVS boundary for the bitstream ashaving a different value of the sum could lead to buffer underruns oroverflows.

Therefore, in an embodiment an indication is carried in the bitstreamthat from a certain point in the bitstream on (e.g. splicing point), thevalue constraint regarding the constant sum of InitCpbRemovalDelay andInitCpbRemovalDelayOffset (and the alternative counterparts) is resetand the sums before and after the point in the bitstream may bedifferent. Unless this indication is present in the bitstream the sumstays constant.

For instance:

When concatenationFlag is equal to 0, it is a constraint of bitstreamconformance that the sum of InitCpbRemovalDelay andInitCpbRemovalDelayOffset is constant across buffering periods.

Otherwise, sum of InitCpbRemovalDelay and InitCpbRemovalDelayOffset doesnot have to be constant across buffering periods. The values ofInitCpbRemovalDelay and InitCpbRemovalDelayOffset are updated to accountfor the arrival times.

In an embodiment, if several bitstreams are spliced, at each splicingpoint a concatenation flag may, e.g., define whether the sum staysconstant or not.

In the following, the fourth aspect of the invention is now described indetail.

In accordance with the fourth aspect of the invention, a video datastream is provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises an indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) indicating whether or not anon-scalable nested picture timing supplemental enhancement informationmessage of a network abstraction layer unit of an access unit of theplurality of access units of a coded video sequence of a one or morecoded video sequences of the video data stream is defined to apply toall output layer sets of a plurality of output layer sets of said accessunit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a value being differentfrom the first value, then the indication does not define whether or notthe non-scalable nested picture timing supplemental enhancementinformation message of said network abstraction layer unit of saidaccess unit applies to all output layer sets of the plurality of outputlayer sets of said access unit.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then saidnetwork abstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then saidnetwork abstraction layer unit does not comprise any other supplementalenhancement information message.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then foreach network abstraction layer unit, which comprises a non-scalablenested picture timing supplemental enhancement information message, ofeach access unit of the plurality of access units of a coded videosequence of the one or more coded video sequences, said networkabstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message, or does not comprise anyother supplemental enhancement information message.

In an embodiment, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then foreach network abstraction layer unit, which comprises a non-scalablenested picture timing supplemental enhancement information message, ofeach access unit of the plurality of access units of each of the one ormore coded video sequences of the video data stream, said networkabstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message, or does not comprise anyother supplemental enhancement information message.

Moreover, a video encoder 100 may, e.g., be provided. The video encoder100 is configured to encode a video into a video data stream. Moreover,the video encoder 100 is configured to generate the video data streamsuch that the video data stream comprises an indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) indicating whether or not anon-scalable nested picture timing supplemental enhancement informationmessage of a network abstraction layer unit of an access unit of theplurality of access units of a coded video sequence of a one or morecoded video sequences of the video data stream is defined to apply toall output layer sets of a plurality of output layer sets of said accessunit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a value being differentfrom the first value, then the indication does not define whether or notthe non-scalable nested picture timing supplemental enhancementinformation message of said network abstraction layer unit of saidaccess unit applies to all output layer sets of the plurality of outputlayer sets of said access unit.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then thevideo encoder 100 is configured to generate the video data stream suchthat said network abstraction layer unit does not comprise any othersupplemental enhancement information message that is different from apicture timing supplemental enhancement information message.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then thevideo encoder 100 is configured to generate the video data stream suchthat said network abstraction layer unit does not comprise any othersupplemental enhancement information message.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then thevideo encoder 100 may, e.g., be configured to generate the video datastream such that for each network abstraction layer unit, whichcomprises a non-scalable nested picture timing supplemental enhancementinformation message, of each access unit of the plurality of accessunits of a coded video sequence of the one or more coded videosequences, said network abstraction layer unit does not comprise anyother supplemental enhancement information message that is differentfrom a picture timing supplemental enhancement information message, ordoes not comprise any other supplemental enhancement informationmessage.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then thevideo encoder 100 may, e.g., be configured to generate the video datastream such that for each network abstraction layer unit, whichcomprises a non-scalable nested picture timing supplemental enhancementinformation message, of each access unit of the plurality of accessunits of each of the one or more coded video sequences of the video datastream, said network abstraction layer unit does not comprise any othersupplemental enhancement information message that is different from apicture timing supplemental enhancement information message, or does notcomprise any other supplemental enhancement information message.

Furthermore, an apparatus 200 for receiving an input video data streamis provided. The input video data stream has a video encoded thereinto.The apparatus 200 is configured to generate an processed video datastream from the input video data stream. Moreover, the apparatus 200 isconfigured to generate the processed video data stream such that theprocessed video data stream comprises an indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) indicating whether or not anon-scalable nested picture timing supplemental enhancement informationmessage of a network abstraction layer unit of an access unit of theplurality of access units of a coded video sequence of a one or morecoded video sequences of the processed video data stream is defined toapply to all output layer sets of a plurality of output layer sets ofsaid access unit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a value being differentfrom the first value, then the indication does not define whether or notthe non-scalable nested picture timing supplemental enhancementinformation message of said network abstraction layer unit of saidaccess unit applies to all output layer sets of the plurality of outputlayer sets of said access unit.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then theapparatus 200 is configured to generate the processed video data streamsuch that said network abstraction layer unit does not comprise anyother supplemental enhancement information message that is differentfrom a picture timing supplemental enhancement information message.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then theapparatus 200 is configured to generate the processed video data streamsuch that said network abstraction layer unit does not comprise anyother supplemental enhancement information message.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then theapparatus 200 may, e.g., be configured to generate the processed videodata stream such that for each network abstraction layer unit, whichcomprises a non-scalable nested picture timing supplemental enhancementinformation message, of each access unit of the plurality of accessunits of a coded video sequence of the one or more coded videosequences, said network abstraction layer unit does not comprise anyother supplemental enhancement information message that is differentfrom a picture timing supplemental enhancement information message, ordoes not comprise any other supplemental enhancement informationmessage.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then theapparatus 200 may, e.g., be configured to generate the processed videodata stream such that for each network abstraction layer unit, whichcomprises a non-scalable nested picture timing supplemental enhancementinformation message, of each access unit of the plurality of accessunits of each of the one or more coded video sequences of the processedvideo data stream, said network abstraction layer unit does not compriseany other supplemental enhancement information message that is differentfrom a picture timing supplemental enhancement information message, ordoes not comprise any other supplemental enhancement informationmessage.

Moreover, a video decoder 300 for receiving a video data stream having avideo stored therein is provided. The video decoder 300 is configured todecode the video from the video data stream. The video data streamcomprises an indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) indicating whether or not anon-scalable nested picture timing supplemental enhancement informationmessage of a network abstraction layer unit of an access unit of theplurality of access units of a coded video sequence of a one or morecoded video sequences of the video data stream is defined to apply toall output layer sets of a plurality of output layer sets of said accessunit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a first value, then thenon-scalable nested picture timing supplemental enhancement informationmessage of said network abstraction layer unit of said access unit isdefined to apply to all output layer sets of the plurality of outputlayer sets of said access unit. If the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has a value being differentfrom the first value, then the indication does not define whether or notthe non-scalable nested picture timing supplemental enhancementinformation message of said network abstraction layer unit of saidaccess unit applies to all output layer sets of the plurality of outputlayer sets of said access unit. The video decoder 300 is configured todecode the video depending on said indication.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then saidnetwork abstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then saidnetwork abstraction layer unit does not comprise any other supplementalenhancement information message. The video decoder 300 is configured todecode the video depending on said indication.

According to an embodiment, for example, if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then foreach network abstraction layer unit, which comprises a non-scalablenested picture timing supplemental enhancement information message, ofeach access unit of the plurality of access units of a coded videosequence of the one or more coded video sequences, said networkabstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message, or does not comprise anyother supplemental enhancement information message.

In an embodiment, e.g., if the indication (e.g., ageneral_same_pic_timing_in_all_ols_flag) has the first value, then foreach network abstraction layer unit, which comprises a non-scalablenested picture timing supplemental enhancement information message, ofeach access unit of the plurality of access units of each of the one ormore coded video sequences of the video data stream, said networkabstraction layer unit does not comprise any other supplementalenhancement information message that is different from a picture timingsupplemental enhancement information message, or does not comprise anyother supplemental enhancement information message.

Furthermore, a system is provided. The system comprises an apparatus 200as described above and a video decoder 300 as described above. The videodecoder 300 is configured to receive the processed video data stream ofthe apparatus 200. Moreover, the video decoder 300 is configured todecode the video from the output video data stream of the apparatus 200.

According to an embodiment, the system may, e.g., further comprise avideo encoder 100. The apparatus 200 may, e.g., be configured to receivethe video data stream from the video encoder 100 as the input video datastream.

In particular, the fourth aspect of the invention relates toconstraining PT SEI to not be paired with other HRD SEIs whengeneral_same_pic_timing_in_all_ols_flag equal to 1

The VVC draft specification comprises a flag calledgeneral_same_pic_timing_in_all_ols_flag in the general HRD parameterstructure which the following semantics:

-   -   general_same_pic_timing_in_all_ols_flag equal to 1 specifies        that the non-scalable-nested PT SEI message in each AU applies        to the AU for any OLS in the bitstream and no scalable-nested PT        SEI messages are present.    -   general_same_pic_timing_in_all_ols_flag equal to 0 specifies        that the non-scalable-nested PT SEI message in each AU may or        may not apply to the AU for any OLS in the bitstream and        scalable-nested PT SEI messages may be present.

In general, when an OLS sub-bitstream is extracted from an originalbitstream (comprises OLS data plus non-OLS data), correspondingHRD-related timing/buffer information for the target OLS in the form ofBuffering Period, Picture Timing and Decoding Unit Information SEImessages which are encapsulated in so called scalable-nesting SEImessages are decapsulated. This decapsulated SEI messages aresubsequently used to replace the non-scalable nested HRD SEI informationin the original bitstream. However, in many scenarios, the content ofsome messages, e.g. the Picture Timing SEI message, may remain the samewhen a layer is dropped, i.e. from one OLS to a sub-set thereof.Therefore, general_same_pic_timing_in_all_ols_flag provides a shortcutso that only BP and DUI SEI messages are to be replaced, but the PT SEIin the original bitstream may stay in effect, i.e. it is simply notremoved during extraction when general_same_pic_timing_in_all_ols_flagis equal to 1. Therefore, no replacement PT SEI message needs to beencapsulated in the scalable-nesting SEI message carrying thereplacement BP and DUI SEI messages and no bitrate overhead isintroduced for this information.

However, in the state-of-the-art, the PT SEI message is allowed to becarried within one SEI NAL unit (NAL unit=network abstraction layerunit) jointly with other HRD SEI messages, i.e. BP, PT and SEI messagesmay all be encapsulated within the same Prefix SEI NAL unit. Hence, anextractor would have to do a deeper inspection of such an SEI NAL unitto understand the comprised messages and when only one of the comprisedmessages (PT) is to be kept during the extraction procedure, it would berequired to practically re-write the show SEI NAL units (i.e. removenon-PT SEI messages). In order to avoid this cumbersome low-levelprocessing and allow an extractor to operate on the non-parameter-setportion of a bitstream entirely on the NAL unit level, it is part of theinvention that a bitstream constraint disallows such bitstreamconstruction. In one embodiment, the constraint is phrased as follows:

-   -   general_same_pic_timing_in_all_ols_flag equal to 1 specifies        that the non-scalable-nested PT SEI message in each AU applies        to the AU for any OLS in the bitstream and no scalable-nested PT        SEI messages are present.    -   general_same_pic_timing_in_all_ols_flag equal to 0 specifies        that the non-scalable-nested PT SEI message in each AU may or        may not apply to the AU for any OLS in the bitstream and        scalable-nested PT SEI messages may be present. When        general_same_pic_timing_in_all_ols_flag equal to 1, it is a        constraint of bitstream conformance that all general SEI        messages in the bitstream containing an SEI message with        payload_type equal to 1 (Picture Timing) shall not contain SEI        messages with payload_type unequal to 1.

In the following, the fifth aspect of the invention is now described indetail.

In accordance with the fifth aspect of the invention, a video datastream is provided. The video data stream has a video encoded thereinto.Moreover, the video data stream comprises one or more scalable nestedsupplemental enhancement information messages. The one or more scalablenested supplemental enhancement information messages comprise aplurality of syntax elements. Each syntax element of one or more syntaxelements of the plurality of syntax elements is defined to have a samesize in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of a portion of thevideo data stream.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements is defined to have the same size inevery one of the scalable nested supplemental enhancement informationmessages of the video data stream or of the portion of the video datastream and in in every one of the non-scalable nested supplementalenhancement information messages of the video data stream or of theportion of the video data stream.

In an embodiment, the video data stream may, e.g., comprise a pluralityof access units, wherein each access unit of the plurality of accessunits may, e.g., be assigned to one of a plurality of pictures of thevideo. The portion of the video data stream may, e.g., be an access unitof the plurality of access units of the video data stream. Each syntaxelement of the one or more syntax elements of the plurality of syntaxelements may, e.g., be defined to have the same size in every one of thescalable nested supplemental enhancement information messages of theaccess unit.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the access unit and in in every one of thenon-scalable nested supplemental enhancement information messages of theaccess unit.

In an embodiment, the portion of the video data stream may, e.g., be acoded video sequence of the video data stream. Each syntax element ofthe one or more syntax elements of the plurality of syntax elements may,e.g., be defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the coded videosequence.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the coded video sequence and in in every one ofthe non-scalable nested supplemental enhancement information messages ofthe coded video sequence.

In an embodiment, each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the video data stream.

According to an embodiment, each syntax element of the one or moresyntax elements of the plurality of syntax elements may, e.g., bedefined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamand in in every one of the non-scalable nested supplemental enhancementinformation messages of the video data stream.

In an embodiment, the video data stream or the portion of the video datastream may, e.g., comprise at least one buffering period supplementalenhancement information message, wherein said buffering periodsupplemental enhancement information message defines the size for eachsyntax element of the one or more syntax elements of the plurality ofsyntax elements.

According to an embodiment, said buffering period supplementalenhancement information message comprises, for defining the size foreach syntax element of the one or more syntax elements of the pluralityof syntax elements, at least one of

-   -   a bp_cpb_initial_removal_delay_length_minus1 element,

a bp_cpb_removal_delay_length_minus1 element,

a bp_dpb_output_delay_length_minus1 element,

a bp_du_cpb_removal_delay_increment_length_minus1 element,

a bp_dpb_output_delay_du_length_minus1 element.

In an embodiment, for each access unit of a plurality of access units ofthe video data stream, which comprises a scalable-nested bufferingperiod supplemental enhancement information message, said access unitmay, e.g., also comprise a non-scalable-nested buffering periodsupplemental enhancement information message.

According to an embodiment, for each single-layer access unit of aplurality of single-layer access units of the video data stream, whichcomprises a scalable-nested buffering period supplemental enhancementinformation message, said single-layer access unit may, e.g., alsocomprise a non-scalable-nested buffering period supplemental enhancementinformation message.

Moreover, a video encoder 100 is provided. The video encoder 100 isconfigured to encode a video into a video data stream. Moreover, thevideo encoder 100 is configured to generate the video data stream suchthat the video data stream comprises one or more scalable nestedsupplemental enhancement information messages. Furthermore, the videoencoder 100 is configured to generate the video data stream such thatthe one or more scalable nested supplemental enhancement informationmessages comprise a plurality of syntax elements. Moreover, the videoencoder 100 is configured to generate the video data stream such thateach syntax element of one or more syntax elements of the plurality ofsyntax elements is defined to have a same size in every one of thescalable nested supplemental enhancement information messages of thevideo data stream or of a portion of the video data stream.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise one or more non-scalable nested supplementalenhancement information messages. The video encoder 100 may, e.g., beconfigured to generate the video data stream such that the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements. The video encoder 100may, e.g., be configured to generate the video data stream such thateach syntax element of the one or more syntax elements of the pluralityof syntax elements may, e.g., be defined to have the same size in everyone of the scalable nested supplemental enhancement information messagesof the video data stream or of the portion of the video data stream andin in every one of the non-scalable nested supplemental enhancementinformation messages of the video data stream or of the portion of thevideo data stream.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that the video data stream may,e.g., comprise a plurality of access units, wherein each access unit ofthe plurality of access units may, e.g., be assigned to one of aplurality of pictures of the video. The portion of the video data streammay, e.g., be an access unit of the plurality of access units of thevideo data stream. The video encoder 100 may, e.g., be configured togenerate the video data stream such that each syntax element of the oneor more syntax elements of the plurality of syntax elements may, e.g.,be defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the access unit.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise one or more non-scalable nested supplementalenhancement information messages. The video encoder 100 may, e.g., beconfigured to generate the video data stream such that the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements. The video encoder 100may, e.g., be configured to generate the video data stream such thateach syntax element of the one or more syntax elements of the pluralityof syntax elements may, e.g., be defined to have the same size in everyone of the scalable nested supplemental enhancement information messagesof the access unit and in in every one of the non-scalable nestedsupplemental enhancement information messages of the access unit.

In an embodiment, the portion of the video data stream may, e.g., be acoded video sequence of the video data stream. The video encoder 100may, e.g., be configured to generate the video data stream such thateach syntax element of the one or more syntax elements of the pluralityof syntax elements may, e.g., be defined to have the same size in everyone of the scalable nested supplemental enhancement information messagesof the coded video sequence.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that the video datastream may, e.g., comprise one or more non-scalable nested supplementalenhancement information messages. The video encoder 100 may, e.g., beconfigured to generate the video data stream such that the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements. The video encoder 100may, e.g., be configured to generate the video data stream such thateach syntax element of the one or more syntax elements of the pluralityof syntax elements may, e.g., be defined to have the same size in everyone of the scalable nested supplemental enhancement information messagesof the coded video sequence and in in every one of the non-scalablenested supplemental enhancement information messages of the coded videosequence.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that each syntax element of the oneor more syntax elements of the plurality of syntax elements may, e.g.,be defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the video data stream.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that each syntaxelement of the one or more syntax elements of the plurality of syntaxelements may, e.g., be defined to have the same size in every one of thescalable nested supplemental enhancement information messages of thevideo data stream and in in every one of the non-scalable nestedsupplemental enhancement information messages of the video data stream.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that the video data stream or theportion of the video data stream may, e.g., comprise at least onebuffering period supplemental enhancement information message, whereinsaid buffering period supplemental enhancement information messagedefines the size for each syntax element of the one or more syntaxelements of the plurality of syntax elements.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that said bufferingperiod supplemental enhancement information message comprises, fordefining the size for each syntax element of the one or more syntaxelements of the plurality of syntax elements, at least one of

a bp_cpb_initial_removal_delay_length_minus1 element,

a bp_cpb_removal_delay_length_minus1 element,

a bp_dpb_output_delay_length_minus1 element,

a bp_du_cpb_removal_delay_increment_length_minus1 element,

a bp_dpb_output_delay_du_length_minus1 element.

In an embodiment, the video encoder 100 may, e.g., be configured togenerate the video data stream such that for each access unit of aplurality of access units of the video data stream, which comprises ascalable-nested buffering period supplemental enhancement informationmessage, said access unit may, e.g., also comprise a non-scalable-nestedbuffering period supplemental enhancement information message.

According to an embodiment, the video encoder 100 may, e.g., beconfigured to generate the video data stream such that for eachsingle-layer access unit of a plurality of single-layer access units ofthe video data stream, which comprises a scalable-nested bufferingperiod supplemental enhancement information message, said single-layeraccess unit may, e.g., also comprise a non-scalable-nested bufferingperiod supplemental enhancement information message.

Furthermore, an apparatus 200 for receiving an input video data streamis provided. The input video data stream has a video encoded thereinto.The apparatus 200 is configured to generate an output video data streamfrom the input video data stream. The video data stream comprises one ormore scalable nested supplemental enhancement information messages. Theone or more scalable nested supplemental enhancement informationmessages comprise a plurality of syntax elements. Each syntax element ofone or more syntax elements of the plurality of syntax elements isdefined to have a same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamor of a portion of the video data stream. The apparatus 200 isconfigured to process the one or more scalable nested supplementalenhancement information messages.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements is defined to have the same size inevery one of the scalable nested supplemental enhancement informationmessages of the video data stream or of the portion of the video datastream and in in every one of the non-scalable nested supplementalenhancement information messages of the video data stream or of theportion of the video data stream. The apparatus 200 is configured toprocess the one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information messages.

In an embodiment, the video data stream may, e.g., comprise a pluralityof access units, wherein each access unit of the plurality of accessunits may, e.g., be assigned to one of a plurality of pictures of thevideo. The portion of the video data stream may, e.g., be an access unitof the plurality of access units of the video data stream. Each syntaxelement of the one or more syntax elements of the plurality of syntaxelements may, e.g., be defined to have the same size in every one of thescalable nested supplemental enhancement information messages of theaccess unit.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the access unit and in in every one of thenon-scalable nested supplemental enhancement information messages of theaccess unit. The apparatus 200 may, e.g., be configured to process theone or more scalable nested supplemental enhancement informationmessages and the one or more non-scalable nested supplementalenhancement information messages.

In an embodiment, the portion of the video data stream may, e.g., be acoded video sequence of the video data stream. Each syntax element ofthe one or more syntax elements of the plurality of syntax elements may,e.g., be defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the coded videosequence.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the coded video sequence and in in every one ofthe non-scalable nested supplemental enhancement information messages ofthe coded video sequence. The apparatus 200 may, e.g., be configured toprocess the one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information messages.

In an embodiment, each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the video data stream.

According to an embodiment, each syntax element of the one or moresyntax elements of the plurality of syntax elements may, e.g., bedefined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamand in in every one of the non-scalable nested supplemental enhancementinformation messages of the video data stream. The apparatus 200 may,e.g., be configured to process the one or more scalable nestedsupplemental enhancement information messages and the one or morenon-scalable nested supplemental enhancement information messages.

In an embodiment, the video data stream or the portion of the video datastream may, e.g., comprise at least one buffering period supplementalenhancement information message, wherein said buffering periodsupplemental enhancement information message defines the size of the oneor more of the plurality of syntax elements. The apparatus 200 may,e.g., be configured to process the at least one buffering periodsupplemental enhancement information message.

According to an embodiment, said buffering period supplementalenhancement information message comprises, for defining the size of theone or more of the plurality of syntax elements, at least one of

a bp_cpb_initial_removal_delay_length_minus1 element,

a bp_cpb_removal_delay_length_minus1 element,

a bp_dpb_output_delay_length_minus1 element,

a bp_du_cpb_removal_delay_increment_length_minus1 element,

a bp_dpb_output_delay_du_length_minus1 element.

In an embodiment, for each access unit of a plurality of access units ofthe video data stream, which comprises a scalable-nested bufferingperiod supplemental enhancement information message, said access unitmay, e.g., also comprise a non-scalable-nested buffering periodsupplemental enhancement information message. The apparatus 200 may,e.g., be configured to process the scalable nested supplementalenhancement information messages and the non-scalable nestedsupplemental enhancement information messages.

According to an embodiment, for each single-layer access unit of aplurality of single-layer access units of the video data stream, whichcomprises a scalable-nested buffering period supplemental enhancementinformation message, said single-layer access unit may, e.g., alsocomprise a non-scalable-nested buffering period supplemental enhancementinformation message. The apparatus 200 may, e.g., be configured toprocess the scalable nested supplemental enhancement informationmessages and the non-scalable nested supplemental enhancementinformation messages.

Moreover, a video decoder 300 for receiving a video data stream having avideo stored therein is provided. The video decoder 300 is configured todecode the video from the video data stream. The video data streamcomprises one or more scalable nested supplemental enhancementinformation messages. The one or more scalable nested supplementalenhancement information messages comprise a plurality of syntaxelements. Each syntax element of one or more syntax elements of theplurality of syntax elements is defined to have a same size in every oneof the scalable nested supplemental enhancement information messages ofthe video data stream or of a portion of the video data stream. Thevideo decoder 300 is configured to decode the video depending on the oneor more syntax elements of the plurality of syntax elements.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of the portion of thevideo data stream and in in every one of the non-scalable nestedsupplemental enhancement information messages of the video data streamor of the portion of the video data stream.

In an embodiment, the video data stream may, e.g., comprise a pluralityof access units, wherein each access unit of the plurality of accessunits may, e.g., be assigned to one of a plurality of pictures of thevideo. The portion of the video data stream may, e.g., be an access unitof the plurality of access units of the video data stream. Each syntaxelement of the one or more syntax elements of the plurality of syntaxelements may, e.g., be defined to have the same size in every one of thescalable nested supplemental enhancement information messages of theaccess unit.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the access unit and in in every one of thenon-scalable nested supplemental enhancement information messages of theaccess unit.

In an embodiment, the portion of the video data stream may, e.g., be acoded video sequence of the video data stream. Each syntax element ofthe one or more syntax elements of the plurality of syntax elements may,e.g., be defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the coded videosequence.

According to an embodiment, the video data stream may, e.g., compriseone or more non-scalable nested supplemental enhancement informationmessages. The one or more scalable nested supplemental enhancementinformation messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements. Each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the coded video sequence and in in every one ofthe non-scalable nested supplemental enhancement information messages ofthe coded video sequence.

In an embodiment, each syntax element of the one or more syntax elementsof the plurality of syntax elements may, e.g., be defined to have thesame size in every one of the scalable nested supplemental enhancementinformation messages of the video data stream.

According to an embodiment, each syntax element of the one or moresyntax elements of the plurality of syntax elements may, e.g., bedefined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamand in in every one of the non-scalable nested supplemental enhancementinformation messages of the video data stream.

In an embodiment, the video data stream or the portion of the video datastream may, e.g., comprise at least one buffering period supplementalenhancement information message, wherein said buffering periodsupplemental enhancement information message defines the size for eachsyntax element of the one or more syntax elements of the plurality ofsyntax elements.

According to an embodiment, said buffering period supplementalenhancement information message comprises, for defining the size foreach syntax element of the one or more syntax elements of the pluralityof syntax elements, at least one of

a bp_cpb_initial_removal_delay_length_minus1 element,

a bp_cpb_removal_delay_length_minus1 element,

a bp_dpb_output_delay_length_minus1 element,

a bp_du_cpb_removal_delay_increment_length_minus1 element,

a bp_dpb_output_delay_du_length_minus1 element.

In an embodiment, for each access unit of a plurality of access units ofthe video data stream, which comprises a scalable-nested bufferingperiod supplemental enhancement information message, said access unitmay, e.g., also comprise a non-scalable-nested buffering periodsupplemental enhancement information message.

According to an embodiment, for each single-layer access unit of aplurality of single-layer access units of the video data stream, whichcomprises a scalable-nested buffering period supplemental enhancementinformation message, said single-layer access unit may, e.g., alsocomprise a non-scalable-nested buffering period supplemental enhancementinformation message.

Furthermore, a system is provided. The system comprises an apparatus 200as described above and a video decoder 300 as described above. The videodecoder 300 is configured to receive the output video data stream of theapparatus 200. Moreover, the video decoder 300 configured to decode thevideo from the output video data stream of the apparatus 200.

According to an embodiment, the system may, e.g., further comprise avideo encoder 100. The apparatus 200 may, e.g., be configured to receivethe video data stream from the video encoder 100 as the input video datastream.

In particular, the fifth aspect of the invention relates to constrainingall BP SEIs messages in a bitstream to indicate the same length ofcertain variable-length coded syntax elements and not be scalable-nestedwithout a non-scalable nested variant in the same AU

The buffering period SEI message, the picture timing SEI message and thedecoding unit information SEI message are used to provide precise timinginformation for the NAL units within a bitstream to control theirtransition through the buffers of a decoder in conformance tests. Somesyntax elements in the PT and DUI SEI message are coded with variablelength and the length of these syntax elements is conveyed in the BP SEImessage. This parsing dependency is a design trade-off. For the cost ofnot allowing PT and DUI SEI message parsing without parsing theassociated BP SEI message first, the benefit of saving sending of thoselength syntax elements at each PT or DUI SEI message is achieved. As theBP SEI message (once per multiple frames) is send much less often thanPT (once per each frame) or DUI SEI messages (multiple times per frame),a bit-saving is achieved through this common design trade-off similar asto how picture header structures can reduce the bit-cost of sliceheaders when many slices are used.

More specifically, the BP SEI message in the current VVC draftspecification includes the syntax elements that are the root of parsingdependencies:

-   -   bp_cpb_initial_removal_delay_length_minus1 that specifies the        coded length of alternative timing initial CPB removal delays of        AUs in the PT SEI message and,    -   bp_cpb_removal_delay_length_minus1 that specifies the coded        length of CPB removal delays and removal delay offsets of AUs in        the PT SEI message and,    -   bp_dpb_output_delay_length_minus1 that specifies the coded        length of DPB output delays of AUs in the PT SEI message and,    -   bp_du_cpb_removal_delay_increment_length_minus1 that specifies        the coded length of the individual and common CPB removal delays        of DUs in the PT SEI message and the CPB removal delays of DUs        in the DUI SEI message and,    -   bp_dpb_output_delay_du_length_minus1 that specifies the coded        length DPB output delays of AUs in the PT SEI message and in the        DU SEI message.

However, a problem arises when a bitstream comprises multiple OLSs.While the BP/PT/DUI SEI messages that apply to the OLS that representsthe bitstream are carried in a verbatim fashion in the bitstream arekeeping track of the parsing dependency is trivial, other pairs ofBP/PT/DUI SEI messages that correspond to the OLSs representing (sub-)bitstreams are to be carried in an encapsulated form in so-calledscalable nesting SEI messages. Still, the parsing dependencies apply andgiven that the number of OLS might be very high, it is a considerableburden for a decoder or parser to keep track of the correct encapsulatedBP SEI message for the sake of the parsing dependency when processingthe encapsulated PT and DUI SEI messages. Especially, since thosemessages can also be encapsulated in different scalable nesting SEImessages.

Therefore, as part of the invention, in one embodiment, a bitstreamconstraint is established that the coded value of the respective syntaxelements describing the lengths must be the same in all scalable-nestedand non-scalable nested BP SEI messages in an AU. Therefore, a decoderor parser only needs to store the respective length values when parsingthe first non-scalable BP SEI message in the AU and can resolve theparsing dependencies of all PT and DUI SEI message in the bufferingperiods that start at the respective AU, whether encapsulated inscalable-nesting SEI messages or not. The following is an example of therespective specification text:

-   -   It is a requirement of bitstream conformance that all        scalable-nested and non-scalable nested buffering period SEI        messages in an AU have the same respective value of the syntax        elements bp_cpb_initial_removal_delay_length_minus1,        bp_cpb_removal_delay_length_minus1,        bp_dpb_output_delay_length_minus1,        bp_du_cpb_removal_delay_increment_length_minus1,        bp_dpb_output_delay_du_length_minus1.

In another embodiment, the constraint is expressed only forscalable-nested BP SEI messages that are in the buffering period thatthe current non-scalable-nested BP SEI message determine as follows:

-   -   It is a requirement of bitstream conformance that all scalable        nested buffering period SEI messages in a buffering period have        the same respective value of the syntax elements        bp_cpb_initial_removal_delay_length_minus1,        bp_cpb_removal_delay_length_minus1,        bp_dpb_output_delay_length_minus1,        bp_du_cpb_removal_delay_increment_length_minus1,        bp_dpb_output_delay_du_length_minus1 then the non-scalable        nested buffering period SEI messages of the buffering period.

Here, the BPs of the bitstream define the scope of the constraints forthe scalable-nested BPs from one scalable nested BP to the next scalablenested BP.

In another embodiment, the constraint is expressed for all AUs of thebitstream, e.g. as follows:

-   -   It is a requirement of bitstream conformance that all        scalable-nested and non-scalable nested buffering period SEI        messages in the bitstream have the same respective value of the        syntax elements bp_cpb_initial_removal_delay_length_minus1,        bp_cpb_removal_delay_length_minus1,        bp_dpb_output_delay_length_minus1,        bp_du_cpb_removal_delay_increment_length_minus1,        bp_dpb_output_delay_du_length_minus1.

In another embodiment, the constraint is expressed only for the AUs in aCVS, so a smart encoder may still be able to facilitate the differencein duration of BPs in the bitstream for the coding of the relevant delayand offset syntax elements. Specification text would be as follows:

-   -   It is a requirement of bitstream conformance that all        scalable-nested and non-scalable nested buffering period SEI        messages in a CVS have the same respective value of the syntax        elements bp_cpb_initial_removal_delay_length_minus1,        bp_cpb_removal_delay_length_minus1,        bp_dpb_output_delay_length_minus1,        bp_du_cpb_removal_delay_increment_length_minus1,        bp_dpb_output_delay_du_length_minus1.

Here, the constraint scope is the CVS.

More specifically, the buffering period or BP SEI message defines aso-called buffering period in which timings of individual pictures usethe picture at the start of a buffering period as an anchor. Thebeginning of a buffering period is instrumental, for instance, to testconformance of random-access functionality in a bitstream.

FIG. 7 illustrates two sets of HRD SEIs, scalable nested SEIs andnon-scalable nested SEIs, in a two-layer bitstream according to anembodiment.

In a multi-layer scenario as shown in FIG. 7 , for instance, thescalable-nested HRD SEIs provide a different buffering period setup(through the BP at POC 0 and POC 3) than the non-scalable nested SEIs(only POC 0) to be used when only layer LO is extracted and played fromPOC3 onwards.

However, this also comes at the increased complexity cost of trackingthe parsing dependencies between the PT and the individual BP messagesas explained above which is undesirable. Therefore, as part of theinvention, in one embodiment, it is prohibited to have scalable-nestedBP SEI message in AUs without a non-scalable nested BP SEI message asfollows:

-   -   It is a requirement of bitstream conformance that no        scalable-nested BP SEI message shall be in an AU that does not        contain a non-scalable-nested BP SEI message.

As the above usage scenario is limited to multi-layer bitstreams, inanother embodiment, the related constraint is limited to single-layerbitstreams as follows:

-   -   It is a requirement of bitstream conformance that no        scalable-nested BP SEI message shall be in a single-layer AU        that does not contain a non-scalable-nested BP SEI message.

Although some aspects have been described in the context of anapparatus, it is clear that these aspects also represent a descriptionof the corresponding method, where a block or device corresponds to amethod step or a feature of a method step. Analogously, aspectsdescribed in the context of a method step also represent a descriptionof a corresponding block or item or feature of a correspondingapparatus. Some or all of the method steps may be executed by (or using)a hardware apparatus, like for example, a microprocessor, a programmablecomputer or an electronic circuit. In some embodiments, one or more ofthe most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software or at leastpartially in hardware or at least partially in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, having electronically readable controlsignals stored thereon, which cooperate (or are capable of cooperating)with a programmable computer system such that the respective method isperformed. Therefore, the digital storage medium may be computerreadable.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein. The data carrier, the digital storagemedium or the recorded medium are typically tangible and/ornon-transitory.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatusor a system configured to transfer (for example, electronically oroptically) a computer program for performing one of the methodsdescribed herein to a receiver. The receiver may, for example, be acomputer, a mobile device, a memory device or the like. The apparatus orsystem may, for example, comprise a file server for transferring thecomputer program to the receiver.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are preferably performed by any hardware apparatus.

The apparatus described herein may be implemented using a hardwareapparatus, or using a computer, or using a combination of a hardwareapparatus and a computer.

The methods described herein may be performed using a hardwareapparatus, or using a computer, or using a combination of a hardwareapparatus and a computer.

The above described embodiments are merely illustrative for theprinciples of the present invention. It is understood that modificationsand variations of the arrangements and the details described herein willbe apparent to others skilled in the art. It is the intent, therefore,to be limited only by the scope of the impending patent claims and notby the specific details presented by way of description and explanationof the embodiments herein.

REFERENCES

-   [1] ISO/IEC, ITU-T. High efficiency video coding. ITU-T    Recommendation H.265 ISO/IEC 23008 10 (HEVC), edition 1, 2013;    edition 2, 2014.

1-188. (canceled)
 189. A video decoder for receiving a video data streamhaving a video stored therein, wherein the video decoder is configuredto decode the video from the video data stream, wherein the video datastream comprises one or more scalable nested supplemental enhancementinformation messages, wherein the one or more scalable nestedsupplemental enhancement information messages comprise a plurality ofsyntax elements, wherein each syntax element of one or more syntaxelements of the plurality of syntax elements is defined to have a samesize in every one of the scalable nested supplemental enhancementinformation messages of the video data stream or of a portion of thevideo data stream, wherein the video decoder is configured to decode thevideo depending on the one or more syntax elements of the plurality ofsyntax elements.
 190. The video decoder according to claim 189, whereinthe video data stream or the portion of the video data stream comprisesat least one buffering period supplemental enhancement informationmessage, wherein said buffering period supplemental enhancementinformation message defines the size for each syntax element of the oneor more syntax elements of the plurality of syntax elements.
 191. Thevideo decoder according to claim 190, wherein said buffering periodsupplemental enhancement information message comprises, for defining thesize for each syntax element of the one or more syntax elements of theplurality of syntax elements, at least one of abp_cpb_initial_removal_delay_length_minus1 element, abp_cpb_removal_delay_length_minus1 element, abp_dpb_output_delay_length_minus1 element, abp_du_cpb_removal_delay_increment_length_minus1 element, abp_dpb_output_delay_du_length_minus1 element.
 192. The video decoderaccording to claim 189, wherein for each access unit of a plurality ofaccess units of the video data stream, which comprises a scalable-nestedbuffering period supplemental enhancement information message, saidaccess unit also comprises a non-scalable-nested buffering periodsupplemental enhancement information message.
 193. The video decoderaccording to claim 190, wherein for each single-layer access unit of aplurality of single-layer access units of the video data stream, whichcomprises a scalable-nested buffering period supplemental enhancementinformation message, said single-layer access unit also comprises anon-scalable-nested buffering period supplemental enhancementinformation message.
 194. The video decoder according to claim 189,wherein the video data stream comprises one or more non-scalable nestedsupplemental enhancement information messages, wherein the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements, wherein each syntaxelement of the one or more syntax elements of the plurality of syntaxelements is defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the video datastream or of the portion of the video data stream and in in every one ofthe non-scalable nested supplemental enhancement information messages ofthe video data stream or of the portion of the video data stream. 195.The video decoder according to claim 192, wherein the video data streamcomprises a plurality of access units, wherein each access unit of theplurality of access units is assigned to one of a plurality of picturesof the video, wherein the portion of the video data stream is an accessunit of the plurality of access units of the video data stream, andwherein each syntax element of the one or more syntax elements of theplurality of syntax elements is defined to have the same size in everyone of the scalable nested supplemental enhancement information messagesof the access unit.
 196. The video decoder according to claim 195,wherein the video data stream comprises one or more non-scalable nestedsupplemental enhancement information messages, wherein the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements, wherein each syntaxelement of the one or more syntax elements of the plurality of syntaxelements is defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the access unitand in in every one of the non-scalable nested supplemental enhancementinformation messages of the access unit.
 197. The video decoderaccording to claim 189, wherein the portion of the video data stream isa coded video sequence of the video data stream, and wherein each syntaxelement of the one or more syntax elements of the plurality of syntaxelements is defined to have the same size in every one of the scalablenested supplemental enhancement information messages of the coded videosequence.
 198. The video decoder according to claim 197, wherein thevideo data stream comprises one or more non-scalable nested supplementalenhancement information messages, wherein the one or more scalablenested supplemental enhancement information messages and the one or morenon-scalable nested supplemental enhancement information messagecomprise the plurality of syntax elements, wherein each syntax elementof the one or more syntax elements of the plurality of syntax elementsis defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the coded videosequence and in in every one of the non-scalable nested supplementalenhancement information messages of the coded video sequence.
 199. Avideo encoder, wherein the video encoder is configured to encode a videointo a video data stream, wherein the video encoder is configured togenerate the video data stream such that the video data stream comprisesone or more scalable nested supplemental enhancement informationmessages, wherein the video encoder is configured to generate the videodata stream such that the one or more scalable nested supplementalenhancement information messages comprise a plurality of syntaxelements, wherein the video encoder is configured to generate the videodata stream such that each syntax element of one or more syntax elementsof the plurality of syntax elements is defined to have a same size inevery one of the scalable nested supplemental enhancement informationmessages of the video data stream or of a portion of the video datastream.
 200. The video encoder according to claim 199, wherein the videoencoder is configured to generate the video data stream such that thevideo data stream or the portion of the video data stream comprises atleast one buffering period supplemental enhancement information message,wherein said buffering period supplemental enhancement informationmessage defines the size for each syntax element of the one or moresyntax elements of the plurality of syntax elements.
 201. The videoencoder according to claim 200, wherein the video encoder is configuredto generate the video data stream such that said buffering periodsupplemental enhancement information message comprises, for defining thesize for each syntax element of the one or more syntax elements of theplurality of syntax elements, at least one of abp_cpb_initial_removal_delay_length_minus1 element, abp_cpb_removal_delay_length_minus1 element, abp_dpb_output_delay_length_minus1 element, abp_du_cpb_removal_delay_increment_length_minus1 element, abp_dpb_output_delay_du_length_minus1 element.
 202. The video encoderaccording to claim 199, wherein the video encoder is configured togenerate the video data stream such that for each access unit of aplurality of access units of the video data stream, which comprises ascalable-nested buffering period supplemental enhancement informationmessage, said access unit also comprises a non-scalable-nested bufferingperiod supplemental enhancement information message.
 203. The videoencoder according to claim 200, wherein the video encoder is configuredto generate the video data stream such that for each single-layer accessunit of a plurality of single-layer access units of the video datastream, which comprises a scalable-nested buffering period supplementalenhancement information message, said single-layer access unit alsocomprises a non-scalable-nested buffering period supplementalenhancement information message.
 204. The video encoder according toclaim 199, wherein the video encoder is configured to generate the videodata stream such that the video data stream comprises one or morenon-scalable nested supplemental enhancement information messages,wherein the video encoder is configured to generate the video datastream such that the one or more scalable nested supplementalenhancement information messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements, wherein the video encoder is configured to generate thevideo data stream such that each syntax element of the one or moresyntax elements of the plurality of syntax elements is defined to havethe same size in every one of the scalable nested supplementalenhancement information messages of the video data stream or of theportion of the video data stream and in in every one of the non-scalablenested supplemental enhancement information messages of the video datastream or of the portion of the video data stream.
 205. The videoencoder according to claim 202, wherein the video encoder is configuredto generate the video data stream such that the video data streamcomprises a plurality of access units, wherein each access unit of theplurality of access units is assigned to one of a plurality of picturesof the video, wherein the portion of the video data stream is an accessunit of the plurality of access units of the video data stream, andwherein the video encoder is configured to generate the video datastream such that each syntax element of the one or more syntax elementsof the plurality of syntax elements is defined to have the same size inevery one of the scalable nested supplemental enhancement informationmessages of the access unit.
 206. The video encoder according to claim205, wherein the video encoder is configured to generate the video datastream such that the video data stream comprises one or morenon-scalable nested supplemental enhancement information messages,wherein the video encoder is configured to generate the video datastream such that the one or more scalable nested supplementalenhancement information messages and the one or more non-scalable nestedsupplemental enhancement information message comprise the plurality ofsyntax elements, wherein the video encoder is configured to generate thevideo data stream such that each syntax element of the one or moresyntax elements of the plurality of syntax elements is defined to havethe same size in every one of the scalable nested supplementalenhancement information messages of the access unit and in in every oneof the non-scalable nested supplemental enhancement information messagesof the access unit.
 207. The video encoder according to claim 199,wherein the portion of the video data stream is a coded video sequenceof the video data stream, and wherein the video encoder is configured togenerate the video data stream such that each syntax element of the oneor more syntax elements of the plurality of syntax elements is definedto have the same size in every one of the scalable nested supplementalenhancement information messages of the coded video sequence.
 208. Thevideo encoder according to claim 207, wherein the video encoder isconfigured to generate the video data stream such that the video datastream comprises one or more non-scalable nested supplementalenhancement information messages, wherein the video encoder isconfigured to generate the video data stream such that the one or morescalable nested supplemental enhancement information messages and theone or more non-scalable nested supplemental enhancement informationmessage comprise the plurality of syntax elements, wherein the videoencoder is configured to generate the video data stream such that eachsyntax element of the one or more syntax elements of the plurality ofsyntax elements is defined to have the same size in every one of thescalable nested supplemental enhancement information messages of thecoded video sequence and in in every one of the non-scalable nestedsupplemental enhancement information messages of the coded videosequence.
 209. A non-transitory computer-readable medium storing a videodata stream, wherein the video data stream has a video encodedthereinto, wherein the video data stream comprises one or more scalablenested supplemental enhancement information messages, wherein the one ormore scalable nested supplemental enhancement information messagescomprise a plurality of syntax elements, wherein each syntax element ofone or more syntax elements of the plurality of syntax elements isdefined to have a same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamor of a portion of the video data stream.
 210. The non-transitorycomputer-readable medium storing the video data stream according toclaim 209, wherein the video data stream or the portion of the videodata stream comprises at least one buffering period supplementalenhancement information message, wherein said buffering periodsupplemental enhancement information message defines the size for eachsyntax element of the one or more syntax elements of the plurality ofsyntax elements.
 211. The non-transitory computer-readable mediumstoring the video data stream according to claim 210, wherein saidbuffering period supplemental enhancement information message comprises,for defining the size for each syntax element of the one or more syntaxelements of the plurality of syntax elements, at least one of abp_cpb_initial_removal_delay_length_minus1 element, abp_cpb_removal_delay_length_minus1 element, abp_dpb_output_delay_length_minus1 element, abp_du_cpb_removal_delay_increment_length_minus1 element, abp_dpb_output_delay_du_length_minus1 element.
 212. The non-transitorycomputer-readable medium storing the video data stream according toclaim 209, wherein for each access unit of a plurality of access unitsof the video data stream, which comprises a scalable-nested bufferingperiod supplemental enhancement information message, said access unitalso comprises a non-scalable-nested buffering period supplementalenhancement information message.
 213. The non-transitorycomputer-readable medium storing the video data stream according toclaim 210, wherein for each single-layer access unit of a plurality ofsingle-layer access units of the video data stream, which comprises ascalable-nested buffering period supplemental enhancement informationmessage, said single-layer access unit also comprises anon-scalable-nested buffering period supplemental enhancementinformation message.
 214. The non-transitory computer-readable mediumstoring the video data stream according to claim 209, wherein the videodata stream comprises one or more non-scalable nested supplementalenhancement information messages, wherein the one or more scalablenested supplemental enhancement information messages and the one or morenon-scalable nested supplemental enhancement information messagecomprise the plurality of syntax elements, wherein each syntax elementof the one or more syntax elements of the plurality of syntax elementsis defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the video data streamor of the portion of the video data stream and in in every one of thenon-scalable nested supplemental enhancement information messages of thevideo data stream or of the portion of the video data stream.
 215. Thenon-transitory computer-readable medium storing the video data streamaccording to claim 212, wherein the video data stream comprises aplurality of access units, wherein each access unit of the plurality ofaccess units is assigned to one of a plurality of pictures of the video,wherein the portion of the video data stream is an access unit of theplurality of access units of the video data stream, and wherein eachsyntax element of the one or more syntax elements of the plurality ofsyntax elements is defined to have the same size in every one of thescalable nested supplemental enhancement information messages of theaccess unit.
 216. The non-transitory computer-readable medium storingthe video data stream according to claim 215, wherein the video datastream comprises one or more non-scalable nested supplementalenhancement information messages, wherein the one or more scalablenested supplemental enhancement information messages and the one or morenon-scalable nested supplemental enhancement information messagecomprise the plurality of syntax elements, wherein each syntax elementof the one or more syntax elements of the plurality of syntax elementsis defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the access unit and inin every one of the non-scalable nested supplemental enhancementinformation messages of the access unit.
 217. The non-transitorycomputer-readable medium storing the video data stream according toclaim 209, wherein the portion of the video data stream is a coded videosequence of the video data stream, and wherein each syntax element ofthe one or more syntax elements of the plurality of syntax elements isdefined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the coded videosequence.
 218. The non-transitory computer-readable medium storing thevideo data stream according to claim 217, wherein the video data streamcomprises one or more non-scalable nested supplemental enhancementinformation messages, wherein the one or more scalable nestedsupplemental enhancement information messages and the one or morenon-scalable nested supplemental enhancement information messagecomprise the plurality of syntax elements, wherein each syntax elementof the one or more syntax elements of the plurality of syntax elementsis defined to have the same size in every one of the scalable nestedsupplemental enhancement information messages of the coded videosequence and in in every one of the non-scalable nested supplementalenhancement information messages of the coded video sequence.